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Overview 


While many incoming community college students and broad-access four-year college 
students are referred to remedial programs in math or English based solely on scores they earn on 
standardized placement tests, large numbers of colleges have begun to use additional measures to 
assess the academic preparedness of entering students. Concomitant with major reform efforts in 
the structure of remedial (or developmental) education coursework, this trend toward the use of 
multiple measures assessment is informed by two strands of research: one suggests that many 
students traditionally assigned to prerequisite remediation would fare better by enrolling directly 
in college-level courses, and the other suggests that different measures of student skills and 
performance, and in particular the high school grade point average (GPA), may be useful in 
assessing college readiness. 


CAPR recently completed a random assignment study of a multiple measures placement 
system that uses data analytics. The aim was to learn whether this alternative system yields 
placement determinations that lead to better student outcomes than a system based on test scores 
alone. Seven community colleges in the State University of New York (SUNY) system 
participated in the study. The alternative placement system we evaluated uses data on prior students 
to weight multiple measures — including placement test scores, high school GPAs, and other 
measures — in predictive algorithms developed at each college that are then used to place 
incoming students into remedial or college-level courses. Nearly 13,000 incoming students who 
arrived at these colleges in the fall 2016, spring 2017, and fall 2017 terms were randomly assigned 
to be placed using either the status quo placement system (the business-as-usual group) or the 
alternative placement system (the program group). The three cohorts of students were tracked 
through the fall 2018 term, resulting in the collection of three to five semesters of outcomes data, 
depending on the cohort. We also conducted research on the implementation of the alternative 
placement system at each college as well as a cost and cost-effectiveness analysis. 


Findings from the implementation and cost components of the study show that: 


e Implementation of the multiple measures, data analytics placement system was 
complex but successfully achieved by all the participating colleges. 


e Because alternative placement resulted in many fewer enrollments in remedial 
courses, the total cost of using the multiple measures system was $280 less 
per student than using the business-as-usual system. 


e Students enrolled in 0.798 fewer credits within three terms under the 
alternative system, saving each student, on average, $160 in tuition/fees. 


Impact findings from the evaluation of student outcomes show that: 


lil 


Many program group students were placed differently than they would have 
been under the status quo system. In math, 16 percent of program group 
students were “bumped up” to a college-level course; 10 percent were 
“bumped down” to a remedial course. In English, 44 percent were bumped up 
and 7 percent were bumped down. 


In math, in comparison to business-as-usual group students, program group 
students had modestly higher rates of placement into, enrollment in, and 
completion (with grade C or higher) of a college-level math course in the first 
term, but the higher enrollment and completion rates faded and then 
disappeared in the second and third terms. 


In English, program group students had higher rates of placement into, 
enrollment in, and completion of a college-level English course across all 
semesters studied. While gains declined over time, through the third term, 
program groups students were still 5.3 percentage points more likely to enroll 
in and 2.9 percentage points more likely to complete a college-level English 
course (with grade C or higher). 


Program group students earned slightly more credits than business-as-usual 
group students in the first and second terms, but the gain became insignificant 
in the third term. No impacts were found on student persistence or associate 
degree attainment. 


All gender, Pell recipient status, and race/ethnicity subpopulations considered 
(with the exception of men in math) had higher rates of placement into college- 
level courses using the alternative system. In English, these led to program 
group course completion rates that, compared to their same subgroup peers, 
were 4.6, 4.5, 3.0, and 7.1 percentage points higher for women, Pell recipients, 
non-Pell recipients, and Black students over three terms. 


Program group students who were bumped up into college-level courses from 
what their business-as-usual placements would have been were 8-10 
percentage points more likely to complete a college-level math or English 
course within three terms. Program group students who were bumped down 
into developmental courses were 8-10 percentage points less likely to 
complete a college-level math or English course within three terms. 


This study provides evidence that the use of a multiple measures, data analytics 
placement system contributes to better outcomes for students, including those from all the 
demographic groups analyzed. Yet, the (relatively few) students who were bumped down into 
developmental courses through the alterative system fared worse, on average, than they would 
have under business-as-usual placement. This suggests that colleges should consider establishing 


placement procedures that allow more incoming students to enroll in college-level courses. 
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Executive Summary 


Placement testing is a near-universal part of the enrollment experience for incoming 
community college students (Bailey, Jaggars, & Jenkins, 2015). Community colleges accept 
nearly all students for admission but then make a determination about whether or not those 
students are immediately ready for college-level coursework. Virtually all community 
colleges (and more than 90 percent of public four-year colleges) use the results of placement 
tests — either alone or in concert with other information — to determine whether students 
are underprepared (Rutschow, Cormier, Dukes, & Cruz Zamora, 2019). Students deemed 
underprepared are typically encouraged or required to participate in remedial coursework 
before beginning college-level courses in those subject areas in which they are found to need 
academic help. 


In recent years, questions have arisen about the efficacy of standardized placement 
tests as well as the utility of traditional developmental coursework. College practitioners and 
others are concerned about whether too many students are unnecessarily required to take 
developmental education courses before beginning college-level work. Traditional 
developmental courses require students to make a substantial investment of time and money, 
and many students who begin college by taking developmental coursework never complete a 
college credential (Bailey et al., 2015). Indeed, research shows that the effects of traditional 
developmental courses are mixed at best (Bailey, 2009; Jaggars & Stacey, 2014). 


Evidence also suggests that the use of placement tests alone is inadequate in 
determining which students need remediation. Studies have shown that the use of multiple 
measures in placement decisions, and in particular the use of high school grade point average 
(GPA), is associated with lower rates of misplacement and higher rates of enrolling in and 
succeeding in college-level courses in math and English (Belfield & Crosta, 2012; Scott- 
Clayton, 2012). Partly in response to these findings, substantial numbers of colleges are 
turning to the use of multiple measures for assessing and placing students. 


In 2015, the Center for the Analysis of Postsecondary Research (CAPR) began work 
on a random assignment study of a multiple measures, data analytics placement system to 
determine whether it yields placement determinations that lead to better student outcomes 
than a system based on test scores alone. The alternative placement system we evaluated uses 
data on prior students to weight multiple measures — including placement test scores, high 
school GPAs, and other measures — in predictive algorithms developed at each college that 
are then used to place incoming students into remedial or college-level courses. Seven 
community colleges in the State University of New York (SUNY) system participated in the 
study: Cayuga Community College, Jefferson Community College, Niagara Community 


College, Onondaga Community College, Rockland Community College, Schenectady 
Community College, and Westchester Community College. A report on early findings from 
this research (Barnett et al., 2018) describes the implementation and costs involved in 
establishing such a placement system as well as the initial effects that using it had on student 
outcomes. The current report shares selected implementation findings but focuses mainly on 
providing impact findings on students during the three semesters following initial placement, 
as well as findings from a cost and cost-effectiveness analysis. A longer-term follow-up 
report on this sample of students is planned for summer 2022. 


Study Design and the Implementation of an Alternative 
Placement System 


Our study compares the effects on student outcomes of placing students into 
developmental or college-level courses using either a multiple measures, data analytics 
placement system or a status quo system that uses just one measure — placement test scores. 
We are also concerned with how the alternative placement system is implemented and with 
its costs. 


Five research questions have guided the study: 


1.How is a multiple measures, data analytics placement system 
implemented, taking into account different college contexts? What 
conditions facilitate or hinder its implementation? 


2. What effect does using this alternative placement system have on 
students’ placements? 


3. With respect to academic outcomes, what are the effects of placing 
students into courses using the alternative system compared with 
traditional procedures? 


4. Do effects vary across different subpopulations of students? 


5. What are the costs associated with using the alternative placement system? 
Is it cost-effective? 


To answer Question |, we conducted two rounds of implementation site visits to each 
of the seven colleges in which we interviewed key personnel, including administrators, staff, 
and faculty. To answer Questions 2 through 4, we tracked eligible students who first began 
the intake process at a participating college in the fall 2016, spring 2017, or fall 2017 term 
through the fall 2018 term. For the analyses presented in this report, student data were 
collected in early 2019 from the seven colleges that participated in the study and from the 
SUNY central institutional research office. The data allowed researchers to observe students’ 


outcomes for three to five semesters following placement, depending on the cohort. To 
answer Question 5, we conducted a study of costs as well as a cost-effectiveness analysis that 
incorporates outcomes data. 


In order to carry out this evaluation, an alternative placement system had to be created 
and implemented, and random assignment procedures had to be established. Researchers and 
personnel at each college collaborated in these activities. We obtained 2—3 years of historical data 
from each college that were then used to create algorithms that weighted different factors 
(placement test scores, high school GPAs, time since high school graduation, etc.) according to 
how well they predicted success in college-level math and English courses. Faculty at each 
college then created placement rules by choosing cut points on each algorithm that would be used 
to place program group students into remedial or college-level math and English courses. 


Extensive effort went into automating the alternative placement system at each 
college so that it could be used with all incoming students. In addition, procedures were 
established to randomly place about half of the incoming students (the program group) using 
the new data analytics system; the other half (the business-as-usual group) were placed using 
each college’s existing placement system (most often using the results of ACCUPLACER 
tests). A total of 12,971 students entered the study in three cohorts. 


Overall, implementation of the multiple measures, data analytics placement system 
created a significant amount of up-front work to develop new processes and procedures that, once 
in place, generally ran smoothly and with few problems. At the beginning of the project, colleges 
underwent a planning process of a year or more, in close collaboration with the research team, in 
order to make all of the changes required to implement the alternative placement system. Among 
other activities, each college did the following: (1) organized a group of people to take 
responsibility for developing the new system, (2) compiled a historical dataset which was sent to 
the research team in order to create the college’s algorithms, (3) developed or improved processes 
for obtaining high school transcripts for incoming students and for entering transcript information 
into IT systems in a useful way, (4) created procedures for uploading high school data into a data 
system where it could be combined with test data at the appropriate time, (5) changed IT systems 
to capture the placement determinations derived from the use of multiple measures, (6) created 
new placement reports for use by students and advisors, (7) provided training to testing staff and 
advisors on how to interpret the new placement determinations and communicate with students 
about them, and (8) conducted trial runs of the new processes to troubleshoot and avoid problems 
during actual implementation. 


While these activities were demanding, every college was successful in overcoming 
barriers and developing the procedures needed to support the operation of the data analytics 
placement system for its students. Five colleges achieved this benchmark in time for 


placement of students entering in fall 2016, while the other two colleges did so in time for 
new student intake in fall 2017. (A fuller account of implementation findings is provided in 
Barnett et al., 2018.) 


Data, Analysis, and Results 


Sample and Method 


In this experimental study, incoming students who took a placement test were 
randomly assigned to be placed using either the multiple measures, data analytics system or 
the business-as-usual system. This assignment method creates two groups of students — 
program group and business-as-usual group students — who should, in expectation, be 
similar in all ways other than their form of placement. We present aggregated findings from 
all participating colleges using data from three cohorts of students who went through the 
placement testing process in the fall 2016, spring 2017, or fall 2017 semester. 


Our final analytic sample consists of 12,971 students who took a placement test at 
one of the seven partner colleges, of which 11,102, or about 86 percent, enrolled in at least 
one course of any kind between the date of testing and fall 2018. Because some students in 
the sample were eligible to receive either a math or an English placement rather than both, 
the sample for our analysis of math outcomes is reduced to 9,693 students, and the sample 
for analysis of English outcomes is reduced to 10,719 students. We find that differences in 
student characteristics and in placement test scores between program group and business-as- 
usual group students are generally small and statistically insignificant, which provides 
reassurance that the randomized treatment procedures undertaken at the colleges were 
performed as intended. 


Our analyses were conducted using ordinary least squares regression models in which 
we controlled for college fixed effects and student characteristics such as gender, 
race/ethnicity, age, and financial aid status, as well as proxies for college preparedness. 


For both math and English, we consider the following outcomes: the rate of college- 
level course placement (versus remedial course placement) in the same subject area, the rate 
of college-level course enrollment in the same subject area, and the rate of college-level 
course completion with a grade of C or higher in the same subject area. Because we might 
expect impacts to change over time, we present impact estimates for one, two, and three 
semesters from testing. (In the full report, we also discuss longer-term outcomes for the first 
cohort of students.) 


Placement Determinations of Program Group Students 


Because the multiple measures, data analytics placement system uses different 
criteria than the business-as-usual system, it could lead to more (or fewer) students being 
placed into college-level math or English courses. Importantly, however, any new placement 
procedure does not change the placement determinations of some students. Figure ES.1 
shows how the placement determinations of program students differed from what they would 
have been under the status quo. As expected, based on prior research, the proportion of higher 
(or “bumped up”) placements outweighed the proportion of lower (or “bumped down’) 
placements in both subject areas but particularly in English, where over half of program group 
students were placed differently than they would have been otherwise. 


Figure ES.1 


Change in Placement Among Program Group Students 


Math English 
= Lower placement (bumped down) @ No change (developmental course) 
= No change (college-level course) = Higher placement (bumped up) 


Main Impact Findings 


As shown in Figure ES.2, placement by the algorithm increased the rate of placement 
into college-level math by 6.5 percentage points. But the associated gains in college-level 
math enrollment and completion were small and short-lived. During the first term, compared 
to business-as-usual group students, program group students were 2.4 percentage points (p < 


.01) more likely to enroll in a college-level math course and 2.0 percentage points (p < .01) 
more likely to pass (with grade C or higher) a college-level math course. The positive impacts 
on both outcomes disappeared by the third term. 


Figure ES.2 
College-Level Math Course Outcomes (Among Students in the Math Subsample) 
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Term 2 


Placement Enrollment Completion 


= Business-as-usual group = Program group 


HD <O1, **p < 05, *p < .10. 


In English we find larger impacts across all outcomes considered. Importantly, these 
positive impacts in English were sustained through the third term after testing. As shown in 
Figure ES.3, program group students’ rate of placement into college-level English was 33.8 
percentage points higher than that of business-as-usual group students. The rates of 
enrollment and completion among program students were also higher. Although business-as- 
usual group students began to catch up with program group students over time, students 
assigned by the algorithm maintained a modest advantage with respect to enrolling in and 
passing college-level English by the end of three semesters. Compared to business-as-usual 
group students, program group students were 5.3 percentage points (p < .01) more likely to 
enroll in a college-level English course and 2.9 percentage points (p < .01) more likely to 
pass (with grade C or higher) a college-level English course through three terms. 


Figure ES.3 


College-Level English Course Outcomes (Among Students in the English Subsample) 


Term 2 Term 2 
Placement Enrollment Completion 
= Business-as-usual group = Program group 


HD < OL, **p < .05, *p < .10. 


In addition to subject-specific impacts, we tested for impacts on overall college-level 
course taking, persistence, and associate degree attainment. Compared to business-as-usual 
group students, program group students earned, on average, 0.35 credits more college-level 
credits one term after testing (p < .01) and 0.31 more credits within the first two terms of 
testing (p < .1), but the gain became insignificant in the third term. The small, early credit 
impact can largely be explained by the algorithm’s effect on college-level course-taking in 
English, suggesting that the benefits of alternative placement did not spill over into other 
subjects. We find no impact on student persistence or associate degree attainment. 


Subgroup Impact Findings 


We also conducted subgroup analyses by gender (female, male), Pell recipient status 
(yes, no), and race/ethnicity (Black, Hispanic, White) on our main outcomes of interest in 
each subject: placement into, enrollment in, and completion of a college-level course. To 
determine whether attainment gaps between subgroups were affected by the multiple 
measures placement system, we also tested the significance of interaction effects between 
treatment status and each subgroup. 


In math, we find higher rates of college-level math placement for all subgroups 
considered except men when placed using the algorithm (p < .05). Our results suggest that 
the alternative placement system reversed placement gaps between female and male students: 
Among students in the business-as-usual group, women were less likely than men to place 
into college-level math; among students in the program group, women were more likely than 
men to place into college-level math. We also find that White students received a larger boost 
into college-level math from alternative placement than did their Black and Hispanic peers; 
that is, among students in the program group, college-level placement gaps between White 
and Black students and between White and Hispanic students grew larger. 


Subgroup analyses in math also show that women, non-Pell recipients, and White 
students in the program group were 3.5, 3.8, and 3.2 percentage points (p < .01), respectively, 
more likely to complete a college-level math course (with grade C or higher) than their same- 
subgroup peers in the business-as-usual group in the term following testing, but these gains 
were not sustained through the second or third terms. We find no evidence that existing course 
completion gaps by Pell recipient status changed as a result of multiple measures placement. 
The male-female completion gap narrowed and the White-Black completion gap widened in 
the first term, but these changes were not sustained in later semesters. 


In English, we find much higher rates of college-level placement (of 30 percentage 
points or more) among program group students versus business-as-usual group students for 
all subgroups considered (p < .01). And we find that use of the alternative placement system 
reversed the difference in the rate of placement into college-level English courses for women 
compared to men and helped to minimize the difference for Black students compared to 
White students. 


We also find that college-level English course completion outcomes for all subgroups 
were higher in the first term when placed using the algorithm (p < .01). These gains faded 
away by the third term for men and for White and Hispanic students, but they did not 
disappear for students in other subgroups. Although their gains declined over time, women, 
Pell recipients, non-Pell recipients, and Black students in the program group were 4.6, 4.5, 
3.0, and 7.1 percentage points more likely than their same-subgroup peers in the business-as- 
usual group to complete a college-level English course (with a grade of C or higher) three 
terms after testing (p < .05 for non-Pell recipients; p < .01 for all others). We do not find any 
evidence that gaps in the rates of course completion between related subgroups changed under 
the alternative placement system. 


Finally, we examined outcomes of program group students whose placement 
determinations changed under the alternative placement system (recall Figure ES.1 showing 
that the placement determinations of only 26 percent of math program students and 51 percent 


of English program students changed from what their business-as-usual placements would 
have been). We find that bumped up students had substantially better outcomes in both math 
and English, and that bumped down students had substantially worse outcomes. Program 
group students who were bumped up into college-level courses from what their business-as- 
usual placements would have been were 8—10 percentage points more likely to complete a 
college-level math or English course within three terms. Program group students who were 
bumped down into developmental courses were 8—10 percentage points less likely to 
complete a college-level math or English course within three terms. 


Our findings also indicate that the college-level pass rates of program group students 
bumped up into college-level courses were very similar to those of students placed under the 
business-as-usual system. Within three terms, the status quo pass rate (with grade C or higher) 
in college-level math was 63 percent; the bumped-up pass rate was 60 percent. The status quo 
pass rate in college-level English was 67 percent; the bumped-up pass rate was 65 percent. 


Cost and Cost-Effectiveness Analysis 


To examine costs, we followed the standard approach for the economic evaluation of 
social programs (Levin et al., 2017). To begin, we itemized all the resources required to 
implement the alternative placement system and the business-as-usual system to calculate direct 
costs. Next we calculated the indirect costs that arise from students taking different pathways 
through college. To calculate cost-effectiveness (from the societal, college, and student 
perspectives), we identified an appropriate measure of effectiveness for each placement system. 
We posited that the total number of college-level credits accumulated in math and English per 
student after three terms would be the most valid measure of effectiveness. 


The cost estimate for the alternative placement system is relative to the cost of 
business-as-usual testing for placement. Relative to the status quo, there are new resource 
requirements for the alternative system with respect to (1) administrative set-up and the 
collecting of data for the placement algorithms in math and English, (2) creating the 
algorithms, and (3) applying the algorithms at the time of placement testing. For both systems, 
there are costs in (4) administering placement tests. We calculated these direct costs for six 
colleges (resource data was insufficient at the seventh college) using the ingredients method 
(Levin et al., 2018). 


Across the six colleges, the total cost to fully implement the new system was 
$958,810 (all costs are presented in present value 2016 dollars) for 5,808 students in a single 
cohort. However, this amount includes the cost of administering placement tests, which is 
estimated to have cost $174,240 for the cohort. Therefore, the net cost of implementing the 
alternative system was $784,560 per cohort, or $140 per student. The cost per student varied 


by college from $70 to $360 per student. This variation is primarily driven by the number of 
students at each college. More enrollments lead to lower costs because the costs of creating 
the algorithm are mostly fixed. Once the alternative placement system became fully 
operational, the ongoing operating costs fell substantially, to $40 per student. 


To determine indirect costs and cost-effectiveness, we use the program effects on 
credits attempted in both developmental and college-level math and English coursework, as 
well as credits earned in college-level math and English courses. Program group students 
enrolled in 1.053 fewer developmental education credits than business-as-usual group 
students — or 30 percent fewer. This represents a substantial savings for both students and 
colleges. But program group students also enrolled in 0.255 more college-level math and 
English credits. In total, students placed under the alternative system attempted 0.798 fewer 
credits (college-level and developmental) than students placed under the status quo. 


While program group students had slightly lower credit completion rates in college- 
level math and English courses compared to business-as-usual group students (62.6 percent 
vs. 63.6 percent), they attempted more college-level courses and earned more college-level 
credits. After three terms, program group students earned 3.975 college-level credits, and 
business-as-usual group students earned 3.874 such credits. Program group students thus 
earned 0.101 more college-level math and English credits. (Although this gain in earned 
credits is not statistically significant relative to business-as-usual group students, it is relevant 
as part of the cost-effectiveness analysis.) 


Indirect costs are the costs of providing all attempted developmental and college- 
level credits in math and English. On average, the cost per developmental credit attempted is 
approximately equal to the cost per college-level credit (developmental classes are typically 
smaller than college-level classes, but faculty pay per class is lower). Funding per credit is 
divided between public support and student tuition/fees; we calculated tuition/fees as 39 
percent of total expenditures per credit. 


The results for this cost-effectiveness analysis from the societal or social perspective 
are shown in Table ES.1. The total cost of the alternative system was $280 less per student 
than the status quo — students took fewer developmental education credits (saving $550) that 
more than offset the direct cost of the alternative placement system and the extra indirect cost 
of providing more attempted college-level credits (at $140 and $130 respectively). The 
alternative placement system is more effective, given 0.101 more college-level credits earned 
after three terms. The cost per earned college-level credit was $1,300 for the business-as- 
usual system and $1,190 for the alternative placement system. 
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Table ES.1 


Cost-Effectiveness Analysis: Social Perspective 


Business-as- Alternative 

Per-student Costs Usual Placement Placement Difference 
Direct cost: Placement $30 $170 $140 
Indirect cost: Attempted developmental $1,820 $1,280 $550 

credits 
Indirect cost: Attempted college-level credits $3,170 $3,300 $130 

in math/English 
Total Cost $5,020 $4,750 —$280 
Earned college-level credits in math/English 3.874 3.975 0.101 
Cost per earned college-level credit $1,300 $1,190 -- 


SOURCES: Tables 4.1 and 4.2; authors’ calculations. Cost figures rounded to nearest 10. 


From the student perspective, the alternative placement system is clearly more 
cost-effective. For students, the only cost was the tuition/fees they paid for credits 
attempted. As students took 0.798 fewer credits under the alternative system, they saved 
$160. However, because students generally do not want to take developmental education, 
it may be more valid to focus on their developmental education savings from the 
alternative system. If students took 1.053 fewer developmental education credits, they 
saved $210 in tuition/fees (4 percent of their total spending on college). 


For colleges, the determination of cost-effectiveness depends on net revenues. 
Colleges must pay to implement the alternative placement system; this additional cost must 
then be recouped by increases in net revenues (revenues over costs) from additional 
coursework. Estimating these costs and revenues at each college is difficult. Nevertheless, 
given that the alternative placement system reduced total costs and increased credit 
accumulation, it is plausible to conclude that it is cost-effective from the college perspective. 


Conclusion and Implications 


Colleges continue to seek ways to give students a good start in their higher education 
journey. The results of this study suggest that using a multiple measures, data analytics 
placement system is one way to increase the opportunity entering students have to succeed in 
college-level coursework. Some more specific lessons from this research are: 


e Single placement tests are not good measures of student readiness to 
undertake college-level courses. As has been shown in other research, we 
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find that high school GPAs, especially in combination with other 
measures, are a better predictor of college course success. 


e Colleges would be wise to set up placement systems that allow more 
students into college-level courses. In this study, students who were on the 
margin of being college-ready were much better off if they were permitted 
to take college courses. This can be accomplished without negatively 
influencing course pass rates. 


e The use of a better placement system is a positive step. However, more is 
needed to improve student outcomes, as the impacts that occurred in this study 
were modest. These can include developmental education reforms as well as 
college-wide approaches to improving student experiences and outcomes. 


This study sheds light on an important way to smooth the road for students entering 
college. Rather than using standardized placement tests alone, colleges can develop and 
deploy a multiple measures assessment and placement system that does a better job of placing 
students into math and English courses at a relatively low cost. The use of such a system, in 
tandem with other initiatives to improve student success, can make a real contribution toward 
improving student success in college. 
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Chapter 1 
Introduction 


Placement testing is a near-universal part of the enrollment experience for incoming 
community college students (Bailey, Jaggars, & Jenkins, 2015). Community colleges accept 
nearly all students for admission but then make a determination about whether or not those 
students are immediately ready for college-level coursework. Virtually all community 
colleges (and more than 90 percent of public four-year colleges) use the results of placement 
tests — either alone or in concert with other information — to determine whether students 
are underprepared (Rutschow, Cormier, Dukes, & Cruz Zamora, 2019). Students deemed 
underprepared are typically encouraged or required to participate in remedial or 
developmental coursework before beginning college-level courses in those subject areas in 
which they are found to need academic help. ! 


In recent years, questions have arisen about the efficacy of standardized placement 
tests as well as the utility of traditional developmental coursework. College practitioners and 
others are concerned about whether too many students are unnecessarily required to take 
developmental education courses before beginning college-level work. Traditional courses 
require students to make a substantial investment of time and money, and many students who 
begin college by taking developmental coursework never complete a college credential 
(Bailey et al., 2015). Indeed research shows that the effects of traditional developmental 
courses are mixed at best (Bailey, 2009; Jaggars & Stacey, 2014). 


Evidence also suggests that the use of placement tests alone is inadequate in determining 
which students need remediation. Studies have shown that the use of multiple measures in 
placement decisions, and in particular the use of high school grade point average (GPA), is 
associated with lower rates of misplacement? and higher rates of enrolling in and passing college- 
level courses in math and English (Belfield & Crosta, 2012; Scott-Clayton, 2012). 


Partly in response to these findings, substantial numbers of colleges are turning to the 
use of multiple measures for assessing and placing students. Findings from a national survey 
conducted by the Center for the Analysis of Postsecondary Readiness (CAPR) in 2016 found 
that between 2011 and 2016 the proportion of colleges that used at least one additional 


' Remedial courses are provided to students who are deemed not ready for college-level math or 
English courses, or for other courses that depend on college-level reading, writing, or numeracy skills. The 
terms developmental education and remedial education are used interchangeably in this report. 

* Misplaced students may be underplaced or overplaced. Underplaced students are enrolled in 
developmental education courses despite having a high probability of being successful in college-level 
courses. Overplaced students are enrolled college-level courses despite having a high probability of failure. 


measure than standardized test scores to assess students’ college readiness increased by 30 
percentage points (Rutschow, Cormier, et al., 2019). The most common alternative measure 
was high school performance, used nationally by 41 percent of community colleges placing 
students in math and 37 percent placing students in English. Multiple measures placement 
systems that use a data analytics approach (such as the system employed by colleges in the 
current study) typically make use of placement test results but also consider other relevant 
data, especially high school GPA. 


The CAPR Assessment Study 


In 2015, CAPR researchers began work on an experimental study of seven State 
University of New York (SUNY) community colleges that used a multiple measures, data 
analytics approach to assessment and placement. The theory of change was straightforward. 
Through more accurate placement, students who would otherwise be underplaced would no 
longer waste time and money in courses that they did not need and that are associated with poorer 
outcomes (Bailey et al., 2015). And those students would avoid becoming discouraged by having 
to take and pay for courses that would not count toward their credential. For students who would 
otherwise be overplaced, more accurate placement would mean that they would not enroll in 
courses for which they were not prepared and which they might fail. Therefore, it was 
hypothesized that students placed using the multiple measures, data analytics system would be 
more successful than students placed using the business-as-usual method. 


Our study employed a randomized controlled trial to find out whether this hypothesis 
was correct. To conduct the study, CAPR initiated a partnership with the State University of 
New York (SUNY) system and seven community colleges: Cayuga Community College, 
Jefferson Community College, Niagara Community College, Onondaga Community College, 
Rockland Community College, Schenectady Community College, and Westchester 
Community College. 


Working with each college, we obtained 2-3 years of historical data that were used to 
create algorithms that weighted different factors (placement test scores, high school GPAs, time 
since high school graduation, etc.) according to how well they predicted success in college-level 
math and English courses. Faculty were then provided with information that would help them 
choose the algorithm cut scores that would be used for placement at each college. 


Extensive effort went into automating an alternative placement system at each college 
so that it could be used with all incoming students. In addition, procedures were established 


3 Among community colleges, the increase was from 27 to 57 percent in math and from 19 to 51 
percent in reading. Among public four-year colleges, the increase was from 27 to 63 percent in math and 
from 15 to 54 percent in reading. 


to randomly place about half of the incoming students (the program group) using the new 
data analytics system; the other half (the business-as-usual group) were placed using each 
college’s existing placement system (most often using cut scores from ACCUPLACER tests). 
A total of 12,971 students entered the study in three cohorts: fall of 2016, spring of 2017, and 
fall of 2017. 


To understand what was involved in deploying the new placement system at the 
participating colleges, site visits were conducted 2—3 times at each college; these allowed us 
to document the implementation process and to better understand the costs involved. To 
assess the initial effects of the alternative placement system, student-level administrative data 
were collected from SUNY and participating colleges in the spring of 2017; these were used 
to provide findings that appear in a prior CAPR study report on implementation and early 
impacts (Barmett et al., 2018). A second round of administrative data was collected in the 
summer of 2019 and includes information on all three cohorts of students, up to five semesters 
after their first enrollment in college. These data are used for the analyses described in the 
current report. 


While the current report shares selected implementation findings (Chapter 2), more 
information on the design and implementation of the multiple measures placement system is 
available in the prior report (Barnett et al., 2018). The current report presents impact findings 
over the first three terms of tracking for all cohorts (Chapter 3). It also provides five-term 
impact findings for the first cohort of students in the sample; presents full-sample findings 
by gender, Pell recipient status, and race/ethnicity subgroups; and discusses the outcomes of 
program group students whose placements changed due to use of the algorithm (all in Chapter 
3). This report also presents findings from a cost and cost-effectiveness analysis (Chapter 4) 
and discusses implications of the study’s results for practice (Chapter 5). We will conduct 
one more round of data collection from the seven participating colleges in this study, 
permitting us to report on the outcomes of students in our sample up to ten semesters 
following placement. A longer-term follow-up report is thus planned for publication in 
summer 2022. 


Summary of Findings 


Findings from the implementation and cost components of the study show that: 


e Implementation of the multiple measures, data analytics placement system 
was complex but successfully achieved by all the participating colleges. 


e Because alternative placement resulted in many fewer enrollments in 
remedial courses, the total cost of using the multiple measures system 
was $280 less per student than using the business-as-usual system. 


Students enrolled in 0.798 fewer credits within three semesters under 
the alternative system, saving each student, on average, $160 in 
tuition/fees. 


Impact findings from the evaluation of student outcomes show that: 


Many program group students were placed differently than they would 
have been under the status quo system. In math, 16 percent of program 
group students were “bumped up” to a college-level course; 10 percent 
were “bumped down” to a remedial course. In English, 44 percent were 
bumped up and 7 percent were bumped down. 


In math, in comparison to business-as-usual group students, program 
group students had modestly higher rates of placement into, enrollment in, 
and completion (with grade C or higher) of a college-level math course in 
the first term, but the higher enrollment and completion rates faded and 
then disappeared in the second and third terms. 


In English, program group students had higher rates of placement into, 
enrollment in, and completion of a college-level English course across all 
semesters studied. While gains declined over time, in the third term, 
program groups students were still 5.3 percentage points more likely to 
enroll in and 2.9 percentage points more likely to complete a college-level 
English course (with grade C or higher). 


Program group students earned slightly more credits than business-as- 
usual group students in the first and second terms, but the gain became 
insignificant in the third term. No impacts were found on student 
persistence or associate degree attainment. 


All gender, Pell recipient status, and race/ethnicity subpopulations 
considered (with the exception of men in math) had higher rates of 
placement into college-level courses using the alternative system. In 
English, these led to program group course completion rates that, 
compared to their same subgroup peers, were 4.6, 4.5, 3.0, and 7.1 
percentage points higher for women, Pell recipients, non-Pell recipients, 
and Black students. 


Program group students who were bumped up into college-level courses 
from what their status quo placements would have been were 8-10 
percentage points more likely to complete a college-level math or English 
course within three terms. Program group students who were bumped 
down into developmental courses were 8—10 percentage points Jess likely 
to complete a college-level math or English course within three terms. 


The study provides evidence that the use of a multiple measures, data analytics 
placement system contributes to better outcomes for students, including those from all the 
demographic groups analyzed. Yet, the (relatively few) students who were bumped down into 
developmental courses through the alterative system fared worse, on average, than they 
would have under status quo placement. This suggests that colleges should consider 
establishing or modifying placement procedures that allow more incoming students to enroll 
in college-level courses. 


About CAPR 


Established in 2014, the Center for the Analysis of Postsecondary Readiness is a 
partnership of research scholars supported by the Institute of Education Sciences, U.S. 
Department of Education, and led by the Community College Research Center (CCRC) at 
Teachers College, Columbia University, and MDRC, a nonprofit research and development 
organization. In addition to the study described here, CAPR has conducted two additional 
major studies, one based largely on a nationally representative survey that provides a 
comprehensive understanding of the landscape of developmental education and reform in 
two- and four-year colleges across the country (Rutschow, Cormier, et al., 2019), and one 
that evaluates an alternative model of developmental math programming that shortens 
students’ time in remediation, tailors content to students’ academic paths, and uses student- 
centered instruction (Rutschow, Sepanik, et al., 2019). CAPR also conducts supplemental 
studies and carries out leadership and outreach activities aimed at improving college 
readiness (postsecondaryreadiness.org). 


Chapter 2 
Placement System and Study Design 


The current study uses a randomized controlled trial to compare the effects on student 
outcomes of placing students into developmental or college-level courses with either a 
multiple measures, data analytics placement system or a status quo system, in place at the 
participating colleges, that used just one measure — placement test scores. In order to carry 
out this evaluation, an alternative placement system had to be created and implemented, and 
random assignment procedures had to be established. Researchers and personnel at each 
college collaborated in these activities. We describe the approach used as well as the broader 
study design in this chapter. 


There are five research questions guiding the study: 


1. How is a multiple measures, data analytics placement system 
implemented, taking into account different college contexts? What 
conditions facilitate or hinder its implementation? 


2. What effect does using this alternative placement system have on students’ 
placements? 


3. With respect to academic outcomes, what are the effects of placing 
students into courses using the alternative system compared with 
traditional procedures? 


4. Do effects vary across different subpopulations of students? 


5. What are the costs associated with using the alternative placement system? 
Is it cost-effective? 


To answer Question |, we conducted two rounds of implementation site visits to each 
of the seven colleges; we spoke with key personnel, including administrators, staff, and 
faculty. To answer Questions 2 through 4, we tracked eligible students who first began the 
intake process at a participating college in the fall 2016, spring 2017, or fall 2017 term 
through the fall 2018 term. These students were randomly assigned to either the program 
group or the business-as-usual group. The original study design called for impact analyses to 
be performed twice — once early in the study, following the end of the first cohort’s first 
semester, and again for all three cohorts following the conclusion of the study’s tracking 
period. For the second set of analyses, which are presented in this report, student data were 
collected in early 2019 from the seven colleges that began participation in the study in fall 
2016, as well as from the SUNY central institutional research unit. This allowed researchers 
to observe students’ outcomes for three to five semesters following placement, depending on 
the cohort. To answer Question 5, we carried out a cost-effectiveness analysis that 


incorporates data collected at the end of the project. One additional round of data will be 
collected and analyzed, resulting in a report that will be released in summer 2022. 


Site Descriptions 


Seven SUNY community colleges participated in this study. Many had a prior interest 
in assessing the effectiveness of their existing placement system before they got involved, 
while others saw participation as an opportunity to improve knowledge and practices in 
student placement. The colleges are diverse in terms of size and population served (see 
Appendix A Table A.1). While the smallest of the colleges serves roughly 5,500 students 
annually, the largest serves nearly 24,000 students. All of the colleges have an open-door 
admissions policy, meaning that they do not have entry requirements for incoming students 
beyond graduating from high school or earning a GED. 


As is common in community college settings, a large portion of students at the 
colleges attend part-time, and many are adult learners, with between 21 and 31 percent of 
students over the age of 25. Most of the colleges serve large numbers of students who receive 
financial aid — more than 90 percent of students receive financial aid at five of the seven 
colleges. The colleges have transfer-out rates of between 18 and 22 percent; their three-year 
graduation rates are between 15 and 29 percent. 


Creation of a Data Analytics Placement System 


Each college took four steps to create a data analytics placement system. They (1) 
used historical data to develop an algorithm for math and English placement, (2) estimated 
historical misplacement rates, (3) used these misplacement rates as the basis for choosing cut 
scores, and (4) installed the new placement system. 


Using Historical Data to Develop Algorithms 


Historical high school and placement test data were needed to create predictive 
algorithms at each college. Five colleges in the study had been using ACCUPLACER tests 
for several years. A sixth college had been using ACCUPLACER tests for English but had 
transitioned from a homegrown math assessment to the ACCUPLACER set of math tests 
more recently; this college is therefore testing the use of the alternative placement system for 
English placement only in this study. The seventh college in our sample had been using 
COMPASS tests, standardized placement tests which were discontinued by the provider 
(ACT) shortly after this study began. This college is also testing the use of the alternative 
system for English placement only. At this college, the predictive algorithm that is being 
tested in the alternative placement system does not make use of any placement test scores; 


rather, it is based only on high school GPA and other high school data. The status quo 
placement system in this case uses only scores from ACCUPLACER, the test that the college 
selected to replace the COMPASS. 


CAPR researchers worked with the appropriate personnel at each college as well as 
SUNY’s central institutional research unit to obtain historical data on students who first 
enrolled during the 2011—12, 2012-13, and 2013-14 academic years. Data on multiple 
measures, such as high school performance and placement test scores, as well as data on 
outcomes in college-level courses were used to create algorithms for predicting student 
performance in college-level math and English among students in the study sample. In some 
instances, data on these measures were available in college systems, stored in digital format. 
Other colleges maintained records of high school transcripts as digital images; in these cases, 
the needed data had to be entered into computer systems by hand. 


In order to estimate the relationships between the measures, or “predictors,” in the 
dataset and performance in an initial college-level course, the historical data used for analyses 
were restricted to students who took placement tests and enrolled in a college-level course 
without first having taken a developmental course. This set of students constituted our 
estimation sample. 


For each of the colleges, we began by creating a model’ for estimating the 
relationship between high school GPA and success (defined as earning a grade of C or 
higher) in an initial college-level course in a given subject, math or English (see Equation 
1 below). We then estimated the relationship between placement test scores and success in 
these initial college-level courses (Equation 2). A third model included both high school 
GPA and placement test scores for the appropriate subject (Equation 3). A fourth model 
added additional information where such information was available (Equation 4). Added 
variables included the number of years that had passed since high school completion and 
whether the student’s diploma was a standard high school diploma or a GED, SAT scores, 
ACT scores, and scores on the New York State Regents Exams? where they were available, 
as well as interaction terms and nonlinear terms for certain variables. Identical procedures 
were followed for both math and English. 


4 We regressed success in a college-level course on various sets of predictors using a linear 
probability model. (Alternative, more intricate models are described by Hastie, Tibshirani, & Friedman, 
2009, but more intricate models we tested yielded similar results.) 

>In order to graduate from high school, students in New York State must pass Regents exams in 
math and English. These exams are intended to measure student achievement in these subjects and are 
typically taken between grades 9 and 12. 


(1) Pr(C or better) = a + (HS GPA)B1 + 

(2) Pr(C or better) = a + (Accuplacer)B1 + € 

(3) Pr(C or better) = a + (HS GPA)B1 + (Accuplacer)B2 + € 

(4) Pr(C or better) = a + (HS GPA)B1 + (Accuplacer)B2 + XB3 + € 


While researchers may look at the individual covariates in a traditional study, the 
focus of this analysis is the overall predictive power of each model. We therefore used the 
Akaike Information Criterion (AIC) to compare the models. The AIC is a measure of model 
fit that combines a model’s log-likelihood with the number of parameters included in the 
model (Akaike, 1998; Burnham & Anderson, 2002; Mazerolle, 2004). When comparing 
models, a lower AIC statistic indicates a better fitting model (Mazerolle, 2004). The best 
fitting model was the one selected for use at each college in the study. Appendix Tables A.2 
and A.3 list the full set of variables used in each college’s algorithm for math and English. 


Estimating Historical Misplacement Rates at Each College 


The data analytics algorithm that was created for each college (in each subject area) 
also allowed us to compute historical underplacement and overplacement rates for math and 
English. We define an underplaced student as one placed into a developmental course who 
could have succeeded in an initial college-level course in the same subject area by earning a 
grade of C or higher.® In conducting analysis on underplacement, a student’s probability of 
succeeding in the college-level course is calculated using the parameters estimated by each 
college’s best fitting model. We define an overplaced student as one unable to pass a college- 
level course who was nonetheless placed into such a course. Importantly, this is not simply 
the inverse of passing with a C or higher, since a D is not considered a failing grade. 
Nonetheless, the model for overplacement uses the same set of predictors selected in 
modeling underplacement. For example, if Equation 4 from above is selected as a college’s 
best fitting model, then each student’s likelihood of failing the initial college-level course is 
calculated using the following equation: 


(5) Pr(Fai/ =a + (HS GPA)Bi + (Accuplacer)B2+ XB3+ € 
® Scott-Clayton (2012), Belfield and Crosta (2012), and Scott-Clayton et al. (2014) used a passing 
grade of B or better as the outcome of interest in their parallel analyses, arguing that this higher threshold 


ensures that only those who are “severely” underplaced will be identified by the model. Given our threshold 
of a grade of C or better, we distinguish our error rates from the rates generated in those prior studies. 
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The overplacement and underplacement rates for each college are the average 
proportion of students with probabilities over or under the college-level cutoff. Appendix 
Table A.4 shows the mean estimated underplacement and overplacement rates for each of the 
seven colleges. 


Choosing Cut Points for Projected Placement and Pass Rates 


After data analytics algorithms were established at each college, we used the 
coefficients from the regressions to simulate placement and success rates as a basis for faculty 
decisions on where to establish cut points that distinguish students ready for college-level 
courses from those needing remediation. Consider the following simplified example using 
Equation 3 from above. Let Y represent the predicted probability of success in a college-level 
course. We use regression coefficients and a student’s own placement test scores and high 
school GPA to predict the probability of earning a C or better in college-level math (Y ) for 
any new student 7. A set of decision rules can then be determined based on these predicted 
probabilities. If the college has one level of developmental math course placement and one 
college-level course placement, the decision rule may be: 


College level if Y, => 0.6 


Placement; = des 
Developmental if Y; < 0.6 


For each college, we generated spreadsheets projecting the share of students who 
would place into a college-level course at any given cut point on Y, as well as the share of 
those students we would anticipate earning a C or better in that course. These spreadsheets 
were given to colleges so that faculty in the relevant departments could set cut points for 
students taking math or English courses.’ The cut point differs from the projected pass rate. 
The cut point represents the lowest probability of passing for any given student; the cut point 
implies that every student must have that probability of passing or higher.’ Many faculty 
opted to create placement rules that either (1) kept pass rates in college-level courses similar 
to historical pass rates or (2) kept college-level placement rates similar to historical placement 


7 See Table 2.1 from Barnett et al. (2018, p. 15) for an example of such a spreadsheet. 

8 For instance, if the cut point were 40 percent, then every student placed into the college-level 
course would need to have a 40 percent chance or greater of passing the college-level course — most 
students would have above a 40 percent chance. This means we should expect the projected pass rate to be 
higher than the cut point. If higher cut points are used — meaning that students must have higher 
probabilities of passing in order to be placed into the college-level course — then the share placed into the 
college-level course declines but the anticipated pass rate increases. 
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rates. Under the first approach, the algorithm tended to predict increases in the number of 
students placed into college-level coursework.’ 


Installing the Alternative Placement System 


Colleges in the study had two options for installing the data analytics placement 
system. At colleges running the system through ACCUPLACER, researchers programmed 
custom rules into the ACCUPLACER software for students selected to be part of the program 
group. The rules specified the ACCUPLACER placement determination for every 
combination of multiple measure values used in the algorithm, which were accessed from a 
pre-registration file created and uploaded with data for each incoming student. Other colleges 
conducted their placement through MDRC’s custom-built server and therefore did not need 
to create a pre-registration file. Instead, student information was sent to MDRC servers in 
one of two ways. Either all information was uploaded together and a placement decision was 
returned for each student, or students’ supplemental information was uploaded in batches and 
test scores were uploaded individually by counselors after students completed their testing. 
The values of the uploaded multiple measures and test scores were then multiplied by their 
respective algorithm weights and summed to generate the predicted probability of success 
and the corresponding placement, which was returned to the college. 


Implementation Findings 


CAPR research teams visited each of the seven participating colleges on two separate 
occasions to learn what college personnel thought about both the status quo and alternative 
placement systems and to better understand the processes required to implement the 
alternative system. 


Overall, implementation of the multiple measures, data analytics placement system 
created a substantial amount of up-front work to develop new processes and procedures that, 
once in place, generally ran smoothly and with few problems. At the beginning of the project, 
colleges underwent a planning process of a year or more, in close collaboration with the 
research team, in order to make all of the changes required to implement the alternative 
placement system. Among other activities, each college did the following: 


° The colleges often used multiple cut points on the range of each algorithm’s student scores to 
place students into different levels of developmental coursework and different levels of college-level 
coursework in math and English. For this study, however, we are considering only two placement 
alternatives: developmental versus college-level placement. Those cut points ranged from a 26 to 73 
percent chance of success in math, and from a 45 to 67 percent chance of success in English. 
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. organized a group of people to take responsibility for developing the new 


system, 


. compiled a historical dataset which was sent to the research team in order 


to create the college’s algorithms, 


. developed or improved processes for obtaining high school transcripts for 


incoming students and for entering transcript information into IT systems 
in a useful way (which in some cases was time-consuming and 
challenging), 


. created procedures for uploading high school data into a data system where 


it could be combined with test data at the appropriate time, 


. changed IT systems to capture the placement determinations derived from 


the use of multiple measures, 


6. created new placement reports for use by students and advisors, 


7. provided training to testing staff and advisors on how to interpret the new 


While these activities were demanding, every college was successful in overcoming 
barriers and developing the procedures needed to support the operation of the data analytics 
placement system for its students. Five colleges achieved this benchmark in time for 
placement of students entering in the fall of 2016, while the other two colleges did so in time 
for new student intake in the fall of 2017. While many interviewees believed that the 
alternative system would place students more fairly and accurately, they also reported 


placement determinations and communicate with students about them, and 


. conducted trial runs of the new processes to troubleshoot and avoid 


problems during actual implementation. 


challenges and concerns. These issues largely involved: 


1 


. undertaking such an extensive reform so quickly and establishing the buy- 


in to do so, 


. obtaining and entering large amounts of high school transcript data into 


the college’s computer system, 


. adjusting classroom and faculty assignments based on changed 


proportions of students in developmental and college-level courses, 


. not having placement information immediately available to students under 


the alternative system (in some cases, students had to wait a day or more 
to get their placement determinations), and 


. the potential limiting of access to support programs intended for 


underprepared (low-placing) students. 
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Randomized Controlled Trial Procedures 


A randomized controlled trial yields the most robust and credible estimates of a 
program’s effects because it makes it possible to determine counterfactual outcomes, that is, 
what would have happened in the absence of the program. To conduct this experimental 
study, our procedures were as follows. First, entering prospective first-year students arrived 
at each college for the intake process. Those with waivers based on SAT scores or with other 
exemptions from both math and English placement testing were not placement tested at all 
but rather went straight into college-level courses; they were not part of the study.'° Before 
taking placement tests, the remaining students (some of whom took tests in only one subject 
area, math or English'') were informed about the research, afforded the opportunity to seek 
additional information, and were able to opt out if they wished.'? Those who continued took 
placement tests and were randomly assigned to be placed using either the status quo method 
(business-as-usual group students) or the method using a multiple measures, data analytics 
algorithm (program group students). 


After taking placement tests, students were notified of their placements into 
developmental or college-level courses either by a college staff member or through an online 
portal, depending on the college. It is important to recognize that 14 percent of students who 
were randomly assigned to the business-as-usual or program group and who received a 
placement later decided not to enroll in any course in the first term after testing. We 
nonetheless include such persons as students for purposes of our intention-to-treat analysis 
and sometimes distinguish “students” from “enrolled students,” those who did enroll in at 
least one course in that first term. The random assignment process was integrated into the 
existing placement procedures at each college, though the way that this was accomplished 
was tailored to individual campuses. Irrespective of the randomization mechanism, business- 
as-usual group students followed status quo placement procedures, and program group 
students were placed using the alternative placement system. Students did not receive 
information on which group they were assigned to. 


10 Students could also be granted waivers after placement testing. In these cases, students’ 
placements were based on the waivers and not their placement scores. 

'! Students who took a placement test only in math were not considered in the analysis of English 
outcomes, and students who took a placement test only in English were not considered in the analysis of 
math outcomes. 

'2 Students who opted out never entered the study; thus, we cannot report on the exact number of them. 
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Chapter 3 
Data, Analysis, and Results 


Data 


The data used to place students and track their outcomes in this study come from two 
main sources: placement records and administrative data from each college. Student-level 
placement records include indicators for students’ actual placement levels in math and 
English, as well as information that is needed to determine students’ placements, regardless 
of assignment to program or business-as-usual group. Placement records from each college 
contain high school GPAs and scores on individual ACCUPLACER tests. Additional 
variables included in the placement records vary by college. Examples of additional variables 
incorporated for certain colleges include the number of years between high school completion 
and college enrollment, type of diploma (high school diploma vs. GED), SAT scores, and 
New York State Regents Exam scores. 


In addition to placement records, college administrative data on students were 
collected. These data include demographic information such as gender, race/ethnicity, age, 
and financial aid status; semesters enrolled; courses taken including course levels; credits 
attempted and earned; and course grades. All participants were tracked from the time of 
testing through fall 2018. 


Sample 


We present aggregated findings using data from three cohorts of students who went 
through the placement testing process at a participating college for the fall 2016 through the 
fall 2017 semesters. Excluded from the sample are students who opted out of the study, those 
who took their first placement test outside of the study intake period, and anyone whose 
ACCUPLACER or writing scores on a college-created test placed them into an English as a 
Second Language (ESL) course. Our final analytic sample consists of 12,971 students who 
took a placement test at one of the seven partner colleges, of which 11,102, or about 86 
percent, enrolled in at least one (developmental or college-level) course of any kind between 
the date of testing and fall 2018. 


Table A.5, located in Appendix A, displays baseline descriptive statistics for students in 
our final analytic sample. Pre-randomization characteristics for the overall sample are reported in 
the first two columns, and additional columns present results for each of the colleges separately. 
On average, students in the sample were more likely to be male (52 percent) and White (43 
percent). Forty-three percent of all students received a federal Pell Grant. 


15 


Table A.5 indicates that there is variation in demographic characteristics across 
colleges. For instance, the percentage of students who identified as White ranged from 81 
percent (College 1) to 24 percent (College 7), while the percentage of students who identified 
as Black ranged from 32 percent (College 6) to 9 percent (College 1). Students who took their 
placement test at College 7 were the more likely to be Hispanic than any other racial/ethnic 
category considered. Using Pell Grant receipt as a proxy for low-income status, average 
family income for study participants also varied across colleges; 60 percent of study 
participants from College 6 received Pell Grants as compared to only 32 percent of students 
from College 5." 


Analytic Method 


To test the hypothesis that a multiple measures, data analytics placement system differs 
from a single test placement system, we compared the average outcomes for students assigned 
to the program and business-as-usual groups using ordinary least squares (OLS) regressions. '4 
As previously discussed in Chapter 2 of this report, CAPR researchers worked with campuses 
to develop college-specific predictive algorithms, specify decision rules, and embed these 
algorithms and rules into regular college placement systems. After the installation of these 
alternative placement systems was complete, incoming students were randomly assigned to be 
placed into math and English courses using either the status quo single placement test system 
(business-as-usual group) or the multiple methods, data analytics system (program group). 
Importantly, the random assignment procedure ensures, in expectation, that students assigned 
to the program group are similar in all ways to those assigned into courses under status quo 
placement rules.!° Any differences in student outcomes observed between groups can thus be 
attributed to the specific placement procedure encountered. 


For a more detailed discussion of the analytic model used in this study, please see 
Appendix B. 


'3 The small proportion of Pell recipients in Colleges 4 and 5 can be explained by the exclusion 
from the sample of students who placed into ESL courses, as these students are substantially more likely to 
be Pell recipients than their peers. 

‘4 Importantly, unless otherwise stated, throughout this report we present intention-to-treat results. 
This means that students who did not enroll in any courses following placement are nonetheless included 
in the sample and were coded with a zero on all enrollment, completion, credit accumulation, persistence, 
and degree attainment outcomes. Therefore, in most cases the impacts shown are based on a sample of 
students that include those who never entered courses, which may understate the impacts on students who 
did in fact enroll. 

‘5 Appendix Table A.6 provides evidence that participants’ demographic and academic 
characteristics, including indicators for missing characteristics, are well balanced across program and 
business-as-usual groups for the final analytic sample. 
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Results 


Changes in Placement 


Program group students whose placements changed under the algorithm were 
more frequently placed into a higher- than a lower-level course. Changes in placement, in 
either direction, were more likely to be observed in English than in math. 


Because the multiple measures placement system uses different criteria than the status 
quo system, it could lead to more (or fewer) students being placed into college-level math or 
English courses. That said, we would expect some students to have the same placement 
regardless of the method used. 


Although most students assigned to the program group were placed into both math 
and English courses based on their subject-specific algorithm scores, in some cases students 
may have not been eligible for placement by the algorithm in one subject or the other. Further, 
at each of the participating colleges, some students were exempt from testing in one or both 
subjects via a waiver system. For example, students scoring at or above a specified threshold 
on a subsection of standardized tests such as the ACT or SAT bypassed placement testing 
requirements and placed directly into college-level courses in the relevant subject.'® 
Additionally, two of the seven participating colleges did not use the alternative system to 
place program students into math courses. Students from these two colleges were excluded 
from math-specific analyses.'’ Of the 6,589 students assigned to the program group, 76 
percent received a math placement based on their algorithm score and 83 percent received an 
English placement based upon their algorithm score. 


Figure 3.1 summarizes changes in placement for program group students who took a 
placement test in each subject.!® Although many program students’ placements did not 
change, 26 percent received a math placement different from what would have been expected 
under the status quo placement rules (i.e., using the existing ACCUPLACER cut scores), and 
51 percent experienced a change in their level of English placement. Importantly, program 
students whose placements changed because of the algorithm include two groups: (1) those 
who were “bumped up” to college-level courses under the alternative placement system but 
who would have been referred to developmental courses under the status quo system, and (2) 


'6 Students from at least one college were also eligible to bypass placement testing requirements 
if they had an A average grade across all their high school courses. 

'7 Among students assigned to the business-as-usual group, 74 percent received a math placement 
and 83 percent received an English placement based on the status quo conditions. 

'8 We consider students who placed into a college-level course coupled with a corequisite 
developmental companion course as having received a developmental placement. 
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those who were “bumped down” to developmental courses under the alternative placement 
system but who would have been referred to college-level courses otherwise. 


Overall, we find that bumped up placements represent the majority of changed 
placements experienced by program students in both subjects. Specifically, among program 
students participating in the math portion of the study, 16 percent were placed into a higher- 
level math course (i.e., a college-level course) than would have been expected under the status 
quo system, and 10 percent were placed into a lower-level math course (i.e., a developmental 
course). Of those participating in the English portion of the study, 44 percent placed into a 
higher-level course, and 7 percent placed into a lower-level course than they would have 
under the status quo system. 


Figure 3.1 


Change in Placement Among Program Group Students 


Math English 
= Lower placement (bumped down) = No change (developmental course) 
= No change (college-level course) = Higher placement (bumped up) 


Students generally followed the placement assignments that they received; 94 percent 
of the 7,361 students who took a math course and 93 percent of the 8,349 students who took 
an English course within the study period took a course matching their placements. Instances 
of students not following their assigned placements can be at least partially explained by the 
fact that we report only on initial placements and therefore do not consider how retesting may 
have changed final placements. 
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Main Effects 


In this section we show the results for each outcome of interest in math and English. 
Outcomes include placement into a college-level course in each subject area, enrollment in a 
college-level course in each subject area, and enrollment in and completion (with grade C or 
higher) of a college-level course in each subject area. !? We also present overall impacts on 
total college-level credits attempted and earned, persistence, and degree completion. Because 
we might expect impacts to change over time, we present impact estimates for one, two, and 
three semesters from testing. All results are calculated using a fully specified model that 
includes college and cohort fixed effects, controls for the set of pre-defined demographic 
characteristics, and controls for students’ academic preparedness using each students’ 
predicted probability of success in a college-level course in both subjects. Results are 
calculated using the full analytic sample; outcomes from all eligible students who took a 
placement test are considered regardless of subsequent college enrollment (i.e., results should 
be interpreted as intent-to-treat estimates).?°? Appendix Tables A.7 through A.24 show 
results for each outcome of interest. 


Math 


Although placement by the algorithm increased the rate of placement into college- 
level math by 6.5 percentage points, gains in college-level math enrollment and completion 
rates were small and short-lived. 


Figure 3.2 displays the treatment effects on placement into, enrollment in, and 
completion (with a grade of C or higher) of college-level math within one, two, and three 
terms, among the 9,693 students who received a math placement. Students in the program 
group were 6.5 percentage points more likely than those in the business-as-usual group to be 
placed into college-level math (p < .01). This impact may also be stated in proportional terms 
by dividing the percentage point difference in outcomes by the business-as-usual group 
outcome. Stated in this way, the multiple measures placement system increased the rate of 
placement into college-level math courses by 18 percent. 


Program students were initially slightly more likely to enroll in or pass college-level 
math, but the positive impact on both outcomes disappeared over time. More specifically, 
program group students were 2.4 (p < .01) and 1.6 (p < .1) percentage points more likely than 
their peers to enroll in college-level math within one and two terms, respectively; however, 


'9 College-level math and English courses include all credit-bearing, transferable courses, 
including gatekeeper courses, that restrict enrollment to students who have successfully completed or been 
exempted from completing all necessary prerequisite courses in the same subject area. 

°° For simplicity’s sake, the figures shown below round results to whole number percentages. In 
the surrounding body text (and in the appendix tables), results are presented to the tenth place. 
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the same students were no more likely to enroll in a college-level math course than their 
business-as-usual peers within three terms. The initial small, positive impact on the 
probability of completing college-level math similarly fades over time. Although students 
placed by the algorithm were 2.0 percentage points or 13 percent more likely to pass college- 
level math in the first term (p < .01), through the second term there were no discernable 
differences between the two groups.”! 


Figure 3.2 
College-Level Math Course Outcomes (Among Students in the Math Subsample) 


44% 46% 48% 


29% 30% 


Term 2 Term 2 


Placement Enrollment Completion 


= Business-as-usual group m@ Program group 


HD < OL, **p < 05, *p < .10. 


English 


Impacts in English were larger than those in math across all outcomes considered. 
Moreover, positive impacts in English were sustained through all three terms, although the 
magnitude of the gains declined over time. 


As shown in Figure 3.3 below, among the 10,719 students who received a placement 
in English, students in the program group were 33.8 percentage points or 73 percent more likely 
than those in the business-as-usual group to be placed in a college-level English course (p < 


21 See Appendix C for a discussion of impacts on college-level math credits attempted and earned. 
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.O1). Program students were also more likely to both enroll in and complete college-level 
English (p < .01). Specifically, program students were 12.7, 6.9, and 5.3 percentage points more 
likely to enroll in a college-level English course within one, two, and three terms of testing, 
respectively. And they were 6.3, 3.3, and 2.9 percentage points more likely to pass (with grade 
C or higher) a college-level English course (p < .01).”” Placement by the algorithm caused an 
additional 158 students to complete college-level English within three semesters. 


Figure 3.3 


College-Level English Course Outcomes (Among Students in the English Subsample) 


Term 2 Term 2 
Placement Enrollment Completion 
= Business-as-usual group = Program group 


*D < O01, **p < .05, *p < .10. 


Credits Attempted and Earned, Persistence, and Degree Attainment 


Placement by the algorithm had no impact on student persistence or associate 
degree attainment. It had a small impact on overall college-level credits earned; this is 
mostly explained by the algorithm’s impact on college-level coursetaking in English. 


In addition to subject-specific impacts, we also test whether placement by the 
multiple measures, data analytics placement system had any impact on overall college-level 
coursetaking, persistence, and degree attainment. We do not find any evidence that the 


22 See Appendix C for a discussion of impacts on college-level English credits attempted and earned. 
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alternative placement system impacted student persistence or associate degree attainment (see 
Figure 3.5). However, as shown in Figure 3.4, students assigned to the program group earned, 
on average, more college-level credits than students in the business-as-usual group in the first 
two terms following testing. Specifically, students assigned by the algorithm earned 0.35 
college-level credits more within one term of testing (p < .01) and 0.31 college-level credits 
more within the first two terms (p < .1). Importantly, these small impacts can be mostly 
explained by the algorithm’s impact on college-level coursetaking in English (see Appendix 
C), suggesting that the benefits of alternative placement did not spill over into other subjects. 


Figure 3.4 


College-Level Credit Outcomes 


Term 2 Term 2 
Attempted Earned 
= Business-as-usual group @ Program group 


RD <O1, *p <.05, *p < 10. 
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Figure 3.5 


Persistence and Degree Attainment 
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Term 2 Term 2 
Persistance Degree attainment 
= Business-as-usual group = Program group 


**ED < O01, **p < 05, *p < .10. (None of the impacts are statistically significant.) 


Longer-Term Effects on the First Cohort 


We have five terms of tracking data for the first cohort of students. Within two 
terms, program group student completion of college-level math courses was no longer 
better than that of the business-as-usual group students in this cohort. However, their 
completion of college-level English courses remained better than their business-as-usual 
group peers in all five terms. 


As previously discussed, the study’s first cohort of students was comprised of those 
who took a placement test in time to be eligible to enroll in classes by fall 2016. Because all 
students were tracked from the time of testing through fall 2018, we are able to track 
outcomes measured up to five terms following testing for the first cohort of students (n = 
4,774). Recall that the study’s first cohort included students from only five of the seven 
participating colleges. This, along with cohort-specific variation, helps explain differences in 
the impact estimates observed between the first cohort presented here and the full analytic 
sample presented above. 
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As shown in Figure 3.6 below, math outcomes from the study’s first cohort are similar 
to those observed for the study’s full sample. While placement by the algorithm increased the 
rate of placement into college-level math by 4.9 percentage points (p < .01), the positive 
impacts on college-level math enrollment and college-level math completion (with grade C 
or higher) were concentrated in the term immediately following testing. More specifically, 
students in the program group were 2.4 percentage points more likely to enroll in college- 
level math (p < .01) and 2.3 percentage points more likely to pass college-level math (p < 
.01) than their peers who were placed under the status quo placement procedure in the first 
term. No statistically significant differences in the rate of college-level math enrollment or 
college-level math completion are observed between program and business-as-usual group 
students after the second and first terms, respectively. 


Figure 3.6 
College-Level Math Course Outcomes for Cohort 1 (Among Students in the Math Subsample) 
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As was the case for the full analytic sample, the impacts on college-level English 
placement, enrollment, and completion, shown in Figure 3.7, are larger than those observed 
for math. Compared to business-as-usual group students, program group students were 34.2 
percentage points or 73 percent more likely to place directly into a college-level English 
course, 17.8 percentage points more likely to enroll in a college-level English course within 
one term of testing, and 11.4 percentage points more likely to complete a college-level 
English course within one term of testing (p < .01). Further, although the positive impact of 
placement by the algorithm decreased over time, students in the program group were more 
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likely than their peers to enroll in and complete a college-level English course in all time 
periods considered. Even through the fifth term, program students in Cohort | were 6.6 
percentage points more likely to enroll in a college-level course and 5.8 percentage points 
more likely to complete a college-level course than their peers (p < .01). 


Figure 3.7 
College-Level English Outcomes for Cohort 1 (Among Students in the English Subsample) 
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We do not find any sustained differences in persistence or degree completion within 
five terms of testing for the first cohort of students. These results are presented in Figure 3.8 
below. (Note that a future report will discuss longer-term results for the full sample of students. ) 
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Figure 3.8 


Persistence and Degree Attainment for Cohort 1 
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Analyses on Gender, Pell Recipient Status, and Race/Ethnicity Subgroups 


To test whether program assignment led to differential intervention effects we conduct 
subgroup analyses by gender, Pell recipient status,”? and race/ethnicity on our main outcomes 
of interest in each subject: placement into, enrollment in, and completion (with grade C or 
higher) of a college-level course. To determine whether rate gaps between subgroups are 
changed by the multiple measures placement system, we also test the significance of interaction 
effects between intervention status and each subgroup (not shown in tables). In other words, 
we test not only whether program students in each subgroup were impacted from placement by 
the algorithm but also whether the subgroups were differentially impacted. Appendix A Tables 
A.25 and A.26 show subgroup results for each outcome of interest. 


3 Importantly, analysis of Pell recipient status is limited to only those students who enrolled in 
any course at the college (“enrolled students”) — a post-random assignment characteristic. As a result, 
these analyses are not causal and may produce biased estimates of treatment effects. 
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Math 


All subgroups except men had higher rates of placement into college-level math 
courses under the alternative placement system. Placement by the algorithm reversed 
placement gaps between men and women but enlarged gaps between White students and 
Black and Hispanic students. In terms of enrollment and completion, there were no 
sustained differences in rates for these outcomes between program and business-as-usual 
group students for any subgroup considered. 


Figures 3.9 to 3.11 present subgroup analyses for each math outcome considered. As 
shown in Figure 3.9, with the exception of men, we find increases in the rate of college-level 
math placement for all subgroups considered when placed using the algorithm (p < .05). 
Furthermore, our results suggest that placement by the algorithm reversed placement gaps 
between female and male students. Among students in the business-as-usual group, women 
were less likely than men to place into college-level math; among students in the program 
group, women were more likely than men to place into college-level math. We also find that 
White program group students experienced a larger gain in terms of placement into college- 
level math than their Black and Hispanic peers. That is, among students in the program group, 
placement gaps between White and Black students and between White and Hispanic students 
grew larger. 


As presented in Figure 3.10, subgroup analyses also show that women, non-Pell 
recipients, and White students placed by the algorithm enrolled in college-level math at a 
higher rate than their same-subgroup peers placed under the business-as-usual procedure 
during the term following testing (p < .01), and that similar gains were sustained through the 
second term for women (p < .1) and through the third term for Pell recipients (p < .1). 
Importantly, although the algorithm did not change enrollment gaps by Pell recipient status 
or race/ethnicity, placement by the algorithm reduced the enrollment gap between women 
and men during the first term; that is, women enrolled in college-level math at a rate closer 
to their male peers under multiple measures placement. This gain, however, was not sustained 
in later terms. 


Although smaller in magnitude, subgroup impacts on college-level math completion 
(Figure 3.11) follow patterns similar to those found for college-level math enrollment. 
Specifically, women, non-Pell recipients, and White students placed by the algorithm were 
3.5, 3.8, and 3.2 percentage points (p < .01), respectively, more likely than their otherwise 
similar peers in the business-as-usual group to complete a college-level math course (with 
grade C or higher) within one term of placement testing. These advantages, however, were 
not sustained in later terms. 
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Further, although there is no evidence that existing completion gaps by Pell recipient 
status changed as a result of multiple measures placement, the male-female completion gap 
narrowed and the White-Black completion gap widened in the first term. These changes in 
completion gaps between student subgroups, however, were not sustained in later semesters, 
meaning that the differences in pass rates (with grade C or higher) between students under 
the multiple measures system were not significantly different than those that existed under 
the status quo system in subsequent terms. 


Figure 3.9 


Placement Into College-Level Math by Subgroup (Among Students in the Math Subsample) 


Males Female No Yes Black Hispanic 
Gender Pell recipient Race/ethnicity 
= Business-as-usual group @ Program group 


*RKD < OL, **p < .05, *p < .10. 
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Figure 3.10 
Enrollment in College-Level Math by Subgroup (Among Students in the Math Subsample) 
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Figure 3.11 
Completion of College-Level Math by Subgroup (Among Students in the Math Subsample) 
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English 


For all subgroups considered, there were positive, sustained, statistically 
significant impacts on college-level English placement, enrollment, and completion (with 
grade C or higher) outcomes under the alternative placement system. Placement by the 
algorithm reversed the placement gap between men and women and narrowed the 
placement gap between White students and Black students. Yet, gaps between gender, Pell 
recipient, and race/ethnicity subgroups were not reduced with regard to enrollment in or 
completion of college-level English courses. 


As shown in Figure 3.12, program group students in each subgroup were at least 30 
percentage points more likely to be placed into a college-level English course than their 
business-as-usual counterparts (p < .01). Importantly, we also find evidence that placement 
by the algorithm changed differences in the rates of placement into college-level English 
between women and men, Pell recipients and non-recipients, and Black and White students; 
placement gaps reversed in favor of women and narrowed in favor of Black students under 
the multiple measures placement system. 


As shown in Figure 3.13, we also find that program group students in all subgroups 
considered were more likely than those placed under the business-as-usual procedure to enroll 
in a college-level English course (p < .01), although the positive impact on enrollment became 
smaller over time. Additionally, we do not find evidence that enrollment gaps changed 
between related subgroups. 


Finally, as shown in Figure 3.14, program group students in each subgroup were more 
likely to pass (with grade C or higher) college-level English within one term of testing as 
compared to their otherwise similar business-as-usual peers (p < .01). Although the benefits 
of being placed by the algorithm became smaller over time, a positive impact on college-level 
English completion remained through three terms for women, Pell recipients, non-Pell 
recipients, and Black students, who maintained a 4.6, 4.5, 3.0, and 7.1 percentage point 
advantage, respectively, over their otherwise similar peers (p < .05 for non-Pell recipients; p 
< .01 for all others). We do not find any evidence that differences in the rates of course 
completion between related subgroups changed under the alternative placement system. 


31 


Figure 3.12 


Placement Into College-Level English by Subgroup (Among Students in the English Subsample) 
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Figure 3.13 
Enrollment in College-Level English by Subgroup (Among Students in the English Subsample) 
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Figure 3.14 
Completion of College-Level English by Subgroup (Among Students in the English Subsample) 
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Analyses on Program Group Students Whose Placements Changed 
Under the Algorithm 


Being “bumped up” to a college-level course by the algorithm had a substantial 
positive effect on students’ academic progress; being “bumped down” to a developmental 
course by the algorithm had a substantial negative effect on students’ academic progress. 
What benefitted students most was not which placement approach was used but rather that 
a student was placed into a college-level course. 


In this section, we analyze what happened to program group students whose 
placements changed due to use of the algorithm (see Figure 3.1 above) compared with 
business-as-usual group students whose placements would have changed had they been 
placed by the algorithm. For each student in the study (whether in the program or business- 
as-usual group), we know what their placements would have been using either the algorithm 
or the status quo procedure. Table 3.1 classifies all students by each possible placement. The 
placements of many students would have been the same under either procedure. *4 Among 
those students whose placements would have differed depending on the placement procedure 
used, we observe four types of students: 


(Type 1) Bumped-up students in the bump-up zone: Program group students who were 
placed into a college-level course but who would have been placed into a 
developmental course under status quo placement. 


(Type 2) Business-as-usual students in the bump-up zone: Students who were placed 
into a developmental course but who would have been placed into a college- 
level course under alternative placement. 


(Type 3) Bumped-down students in the bump-down zone: Program group students who 
were placed into a developmental course but who would have been placed into 
a college-level course under status quo placement. 


(Type 4) Business-as-usual students in the bump-down zone: Students who were placed 
into a college-level course but who would have been placed into a 
developmental course under alternative placement. 


>4 Tt is important to recognize that the alternative placement procedure that was used for program 
group students did not change course placements for many students: 74 percent of program group students 
in math and 49 percent of program group students in English received the same placement under the 
alternative placement system that they would have received under the status quo system. 
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Table 3.1 


Placement Zones for All Students 


Actual Placement 


Placement Zone 


Program Group 


Business-as-Usual Group 


BUMP-UP ZONE 


College-level 


(Type 1) 


Math: n = 814 
English: n = 2,415 


Developmental 


(Type 2) 


Math: n = 789 
English: n = 2,249 


NO CHANGE ZONE 
(College-level) 


College-level 


Math: n = 1,372 
English: n = 2,008 


College-level 


Math: n = 1,281 
English: n = 1,941 


NO CHANGE ZONE 
(Developmental) 


Developmental 


Math: n = 2,312 
English: n = 647 


Developmental 


Math: n = 2,175 
English: n = 713 


BUMP-DOWN ZONE 


Developmental 


(Type 3) 


Math: n = 489 
English: n = 365 


College-level 


(Type 4) 


Math: n = 461 
English: n = 381 


Students in the bump-up zone (Types | and 2) had algorithm scores that met or 
exceeded their college’s minimum acceptable likelihood of succeeding in a college-level 
course; bumped-up students (Type 1) are program group students who would have been placed 
into developmental education under the business-as-usual procedure but who were instead 
placed into a college-level course. Students in the bump-down zone (Types 3 and 4) had 
algorithm scores that fell below their college’s minimum acceptable likelihood of succeeding 
in a college-level course; students who were bumped down (Type 3) are program group 
students who would have been placed into a college-level course under the business-as-usual 
procedure but who were instead placed into a developmental education course. Here we explore 
the effects of being bumped up and bumped down for these two subpopulations of students. 
Appendix Tables A.27 through A.32 show subgroup results for each outcome of interest. 
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Student Performance Under the Algorithm 


We first undertake a preliminary analysis to determine whether the algorithm was 
better than the status quo procedure at predicting college-level course success. We compare 
the outcomes of program group students whose placement was bumped up as a result of the 
algorithm (Type | from above) to the outcomes of business-as-usual students in the bump- 
down zone whose predicted probability of success in the college-level course by the algorithm 
was lower than the threshold chosen by their college (Type 4 from above). Both groups of 
students actually placed into college-level courses. In math, we find that a smaller proportion 
of the latter group than the former group passed college-level math (with grade C or higher) 
(49 percent vs. 60 percent of those who enrolled in the course). In English, there was a similar 
difference (51 percent vs. 65 percent of those who enrolled in the course). This suggests that 
the algorithm was better than the status quo procedure at predicting success.”> Yet it is also 
the case that the algorithm was not designed to account for the probability of success in 
college-level courses after placement into developmental education. The consequences of this 
limitation are explored below. 


Effects on Students in the Bump-Up Zone 


All students represented in Figures 3.15 and 3.16 are program and business-as-usual 
group students in the bump-up zone. All of them had placement test scores that would have 
placed them into developmental courses under the status quo system. However, all students 
also had algorithm scores (based on placement test scores as well as other measures such as 
high school GPA) that made their probability of success exceed the algorithm cutoffs at their 
colleges. This means that, in this analytic subsample, all program group students should have 
been bumped up into college-level courses and all business-as-usual students should have 
been placed into developmental courses, which occurred in most cases.”° 


Students in the bump-up zone for math are shown in Figure 3.15. Through the third 
term, 37.3 percent of students in the business-as-usual group had enrolled in college-level 
math.”’ In the program group, the rate of college-level coursetaking was substantially higher: 


°5 The predicted probability of success generated by the algorithm for these groups had a wider 
difference than the observed outcomes. The predicted probabilities were 60 percent vs. 35 percent for math 
and 73 percent vs. 51 percent for English. 

6 A small percentage of students were incorrectly placed, as shown in the first panel of Figures 
3.15 and 3.16. 

27 About 5 percent of students in the business-as-usual group took a college-level math course 
within one term of testing (despite not being placed into a college-level course based on their initial 
placement test results). In math, retesting or overrides most frequently explain such discrepancies. Very 
few students in our overall sample and zero business-as-usual students in the bump-up zone for math placed 
into a corequisite math course sequence, which pairs a developmental course with a college-level course in 
the same term. 
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44.1 percent took a college-level math course within one term and 59.2 percent did so within 
three terms. Since more program group students enrolled in a college-level course, it is not 
surprising that more of them than business-as-usual students completed it. Specifically, 
program group students were 20.6 percentage points more likely to pass college-level math 


(with grade C or higher) within one term and 9.6 percentage points more likely to do so within 
three terms. 


Figure 3.15 
College-Level Math Outcomes Among Students in the Bump-Up Zone 


37% 


30% 
271% 24% "26% 
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Term | Term 2 Term 3 Term 1 Term 2 Term 3 
Placement Enrollment Completion 
= Business-as-usual group = Program group 


KD < OL, **p < .05, *p < .10. 
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Figure 3.16 


College-Level English Outcomes Among Students in the Bump-Up Zone 


Term 2 Term 2 
Placement Enrollment Completion 
= Business-as-usual group = Program group 


*D < OL, **p < .05, *p < .10. 


Figure 3.16 shows that the patterns of impacts for students in the bump-up zone for 
English are similar to those observed among students in the bump-up zone for math. 
However, in the first term, 27.8 percent of business-as-usual group students took a college- 
level English course despite being placed into a developmental course.*® This can be 
explained by at least two factors. First, we consider students who placed into a college-level 
course coupled with a corequisite developmental companion course as having received a 
developmental placement, and we expect students who received such a placement to have 
simultaneously enrolled in a developmental and a college-level course. Of the 627 business- 
as-usual group students in the English bump-up zone who received a developmental 
placement and took a college-level course in the first term, 322, or about half, were enrolled 
in a corequisite developmental English companion course. Second, because we define a 


8 Only 54 percent of business-as-usual group students in the English bump-up zone took a 
developmental English course within one term, even though they were placed into developmental English. 
Of these students, only 35 percent passed the developmental course. This has important implications for 
the longer-term impacts of this study. More specifically, it sets an upper bound on the number of business- 
as-usual students who could enroll in and pass a college-level English course in the subsequent term. In 
math, only 67 percent of the business-as-usual group took their assigned developmental math course within 
one semester, with just 44 percent of those passing it. 
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student’s placement level using initial placement test results, it is possible for a student’s final 
placement to change due to retesting or overrides granted by an advisor or faculty member. 


By the third term, 57.3 percent of business-as-usual group students in the bump-up zone 
had enrolled in college-level English. In the program group, 72.7 percent had enrolled in 
college-level English. A greater proportion of program than business-as-usual group students 
completed it, too (with a difference of 8.7 percentage points more through the third term). 


Overall, college credit accumulation is also greater for those program group students 
bumped up into college-level courses. Students bumped up in both subjects earned 4 more 
college credits than their counterparts within three terms. We find no differences in 
persistence or degree attainment in the same timeframe. 


Effects on Students in the Bump-Down Zone 


All students represented in Figures 3.17 and 3.18 had placement scores that would 
have placed them into college-level courses. However, all these students also had high school 
GPAs or scores on other measures that made their probability of success lower than the math 
or English algorithm cutoff at their college. This means that in this analytic subsample, 
program students were bumped down into developmental courses while business-as-usual 
students were placed into college-level courses.”? 


°° A small percentage of students were incorrectly placed, as shown first panel of Figures 3.17 and 3.18. 
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Figure 3.17 
College-Level Math Outcomes Among Students in the Bump-Down Zone 
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Figure 3.17 shows outcomes of students in the bump-down zone for math. Students 
in the program group, i.e., those who were bumped down into developmental math, were 
much less likely to enroll in college-level math than their counterparts, even through three 
semesters (which allows time for completing developmental courses). By the end of three 
terms, program group students were 21.0 percentage points less likely to have enrolled in 
college-level math, and they were 8.7 percentage points less likely to have completed (with 
grade C or higher) college-level math than their peers who were placed directly into a college- 
level course under business-as-usual conditions. For these students, the algorithm determined 
that they were relatively unlikely to succeed in a college-level course if placed directly into 
it. However, what the algorithm failed to consider is that they were even less likely to succeed 
in a college-level course if placed into developmental education. 
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Figure 3.18 
College-Level English Outcomes Among Students in the Bump-Down Zone 
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HD < OL, **p < 05, *p < .10. 


Figure 3.18 shows outcomes of students in the bump-down zone for English. We find 
that through three terms program group students who were bumped down into a 
developmental English course were 16.3 percentage points less likely to enroll in college- 
level English and 7.5 percentage points less likely to pass a college-level course than their 
business-as-usual counterparts. 


Despite the fact that their algorithm scores suggested a relatively poor prognosis of 
success in college-level courses, developmental course placements did not improve program 
students’ outcomes. Instead, alternative placement lowered bumped-down students’ chance 
of success relative to their peers who placed directly into college-level courses under the 
status quo placement system. On average, when taking into account course enrollments, 
progression, and persistence, these students would have been better off placing directly into 
college-level courses. 


Effect Sizes on Bumped-Up and Bumped-Down Students 


Two related observations arise from the analysis of students in the bump-up and 
bump-down zones. The first is that the negative effect of bumping down students is similar 
in magnitude to the positive effect of bumping up students (around 8—10 percentage points 
on completing the college-level course). Despite the usefulness of the algorithm at predicting 
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outcomes (i.e., students’ likelihood of passing college-level math if placed directly into it), it 
is far less useful, if useful at all, at predicting impacts (i.e., students’ likelihood of passing a 
college-level course if placed directly into it vs. if placed into a developmental course). The 
students bumped down by the algorithm would have benefitted as much from college-level 
placement as those it bumped up. 


The second observation is that while we find very different impacts across subjects 
in the full sample analysis, the impacts are of similar magnitude in math and English when 
the analysis is conducted on these subgroups of students in the bump-up and bump-down 
zones. Bumping up students in math was just as effective as bumping up students in English, 
and bumping down students had equivalent negative effects in both subjects. 


College-Level Pass Rates of Bumped-Up Students 


Faculty considering a modification to traditional placement may be concerned that 
bumped-up students will lower the pass rates in their college-level courses. This does not 
appear to be a major concern. Dividing the percentage of students passing a course by the 
percentage of students enrolling in the same course yields its pass rate. Among those in the 
program group who were bumped up, 59 percent took the college-level math course and about 
35 percent passed it (with grade C or higher) within three terms. This yields a 60 percent pass 
rate in math (35 percent out of 59 percent). The same calculation yields an English pass rate 
of 65 percent (48 percent out of 73 percent). 


A representation of what might be perceived by instructors as the status quo pass rate 
can be calculated from the entire business-as-usual group sample of students placed directly 
into college-level courses as well as those who took them after developmental courses. 
Compared to the status quo’s 63 percent pass rate in math and 67 percent pass rate in English, 
the bumped-up pass rate is 3 percentage points lower for math and 2 percentage points lower 
for English within three terms. Bumped up students passed their classes at almost the same rate 
as traditional students, not at the dramatically lower rates as is sometimes feared by faculty. 
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Chapter 4 
Cost and Cost-Effectiveness Analysis 


At present, there is only limited evidence on the economics of reforms to 
developmental education, and there is no evidence on the economic value of more accurate 
placement systems.*° In theory, more accurate placement has the potential to be both cost- 
effective and efficient. Improved placement accuracy eliminates developmental education 
courses entirely for students who are correctly placed into college-level courses.*! These 
students save on tuition and fees; and colleges no longer allocate resources to unnecessary 
remedial coursework. 


Here we perform an economic evaluation of the multiple measures, data analytics 
placement system as implemented at the seven community colleges in this study. An initial 
evaluation included a calculation of the costs of implementing this alternative placement 
system (Barnett et al., 2018); the current evaluation includes identification of the 
effectiveness of alternative placement. We use this evidence to calculate the total resource 
consequences of alternative placement and to perform cost-effectiveness analysis. We 
calculate baseline results (and undertake sensitivity testing). All results are derived from 
comparing the alternative placement system with the business-as-usual placement system 
used at the colleges. Together, these analyses establish whether the alternative system is 
affordable, cost-effective, and efficient for students, colleges, and society. 


This chapter is structured as follows. First, we describe the applied economic 
methods. Second, we summarize the relevant evidence from Barnett et al. (2018) and the 
current report. Third, we calculate results for the economic metrics of alternative placement 
in terms of resource consequences and incremental cost-effectiveness. Finally, we provide an 
overall conclusion on the economic case for replacing traditional placement systems with 
multiple measures, data analytics placement systems. 


Method 


For this economic evaluation we follow the standard approach for social programs 
(Levin et al., 2017). To begin, we establish the policy alternatives: assigning students using 


3° On the efficiency of corequisite remediation see Jenkins et al. (2010). Boatman (2012) reported 
on possible cost savings from alternative redesigns of developmental education in Tennessee. Belfield et 
al. (2014) identified large efficiency gains if more students enter college ready without needing remediation 
(see also Belfield et al., 2016). Hemelt et al. (2018) explored general differences in costs per course. 

3! By contrast, an alternative developmental education reform — corequisite remediation — may 
be high in cost if it increases the course loads of students who are assigned to developmental education. 
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business-as-usual placement versus alternative placement.*” Here, we are primarily interested 
in the economic case for the alternative placement system from the societal perspective. 


Next, we itemize all the resources required for the two placement systems. This will 
tell us how affordable each placement system is. For educational interventions, there are direct 
and indirect costs. Direct costs are the costs of implementing the respective placement systems. 
Indirect costs are the cost consequences that arise from taking different pathways through 
college. For example, if the alternative placement system increases enrollment in college 
courses, then that will cause students to pay more in tuition/fees; colleges will also have to 
provide more courses. These indirect costs are important because these placement systems 
differentially affect both the numbers and types of courses that students take during their time 
in college. These cost consequences may even exceed the direct implementation costs. 


We then identify an appropriate measure of effectiveness for each system. In 
conjunction with data on costs, this allows us to perform cost-effectiveness analysis and 
compare business-as-usual with alternative placement. For the purposes of this evaluation, 
we posit that total college-level credits accumulated in math and English through three terms 
is the most valid measure of effectiveness. Credits in math and English are needed in order 
to complete a credential at community college. College credits are a continuous outcome 
(more credits are better) and they are broadly proportionate (each extra credit is valued to the 
same extent). Although credential completion may be considered the ideal measure of 
effectiveness, credits lead directly to completion.** Importantly, while alternative placement 
affects the number of developmental education courses taken as well, developmental credits 
or developmental course pass rates are not considered valid effectiveness outcomes for this 
economic analysis as they are not building blocks leading to a credential. 


Evidence 


Implementation Costs of Alternative Placement 


Evidence is available on how each college implemented their alternative placement 
system (Barnett et al., 2018). Alternative placement required an initial investment in time and 
IT resources followed by (more modest) resources for ongoing operation. The cost estimate 
for alternative placement is relative to the cost of business-as-usual placement. Relative to 


3? Business-as-usual is typically assignment based on a standardized test. See Barnett et al. 
(2018) for a full description of the specific alternative placement system at each college. 

33 Evidence on completion is not available for the sample of students in this evaluation; also, 
community college students are enrolled in different programs (e.g., for certificates and degrees), and 
many seek to transfer to obtain a four-year degree. Our analysis yields very similar results as an 
alternative measure of effectiveness: number of courses taken. 
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business-as-usual placement, new resource requirements were needed for alternative 
placement with respect to: (1) administrative set-up and collecting data for the placement 
algorithms in math and English); (2) creating the algorithms for implementation; and (3) 
applying the algorithms at the time of placement testing. For both placement systems there 
were costs in (4) administering placement tests. 


Evidence on the direct costs of both placement systems is given in Table 4.1. The 
costs are calculated for six colleges using the ingredients method (Levin et al., 2017).*4 Costs 
are derived from the inputs used at each college, multiplied by standardized prices per input. 
All costs are expressed in present value 2016 dollars. These cost estimates should be 
interpreted as the expected cost of implementing an alternative placement system at a college 
of similar size and organization as the six sample colleges. (Initial implementation costs are 
amortized over multiple cohorts). 


Table 4.1 


Direct Implementation Costs 


Range Per College 
Total Lower Bound College | Upper Bound College 
Students per semester 5,808 2,750 505 
Total Placement Cost: 
Alternative placement $958,810 $268,890 $196,170 
Busmesy-asuyual $174,240 $82,590 $15,150 
placement 
Alternative placement 
incremental cost: 
Per semester $784,560 $186,300 $181,020 
Per student $140 $70 $360 


SOURCE: Barnett et al. (2018), Table 5.1, with one additional college. 

NOTES: 2016 dollars. Present values (d = 3%). Rounded to $10. Ingredients information on FTEs from interviews 
with key personnel at six colleges. Cost data not available for one college. Costs amortized over cohorts, see Barnett 
et al. (2018). Student cohorts rounded to nearest 10. 


Across the six colleges, the total cost to fully implement the alternative system was 
$958,810 across 5,808 students in a single cohort. However, this amount includes the cost of 
administering placement tests, which was also incurred under the business-as-usual system. 


34 Resource data was insufficient at the seventh (remaining) college. With personnel changes, 
information on resource use was not available for all ingredients. Costs could not therefore be estimated 
with precision. Based on available information on ingredients, it is unlikely that unit costs for this college 
exceeded the average across the remaining six colleges. 
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Business-as-usual placement is estimated to have cost $174,240 for the cohort. Therefore, the 
net cost of implementing alternative placement was $784,560 per cohort or $140 per student. 


The cost per student varied across each college. The upper and lower bound costs 
across the six colleges were between $70 and $360. This variation is primarily driven by the 
number of students at each college. More enrollments lead to lower costs because the costs 
of creating and implementing the algorithms are mostly fixed, i.e., they do not vary with the 
number of students involved. 


Table 4.1 reports annual implementation and operating costs during the first five 
years of the alternative system. Once alternative placement is fully operational, then initial 
implementation costs are no longer relevant. After this time, the operating cost fell 
substantially, to $40 per student. (If the alternative system were to be adopted more quickly 
than at the sample colleges, costs would be lower than $140; if colleges were to abandon 
alternative placement before five years, costs would be higher than $140.) 


Program Impacts for Alternative Placement 


The relevant program impacts are given in Table 4.2. Program group students 
attempted 1.053 fewer developmental education credits on average than did business-as-usual 
group students; in total, developmental course enrollments were lower by 30 percent. 
Program group students attempted 0.255 more college-level credits in math and English; in 
total, attempted college-level credits were higher by 4 percent. Thus, the net effect on 
attempted credits is moderate. In total, program group students attempted 0.798 fewer credits 
(college-level and developmental) than business-as-usual group students. 


Table 4.2 
Impacts on Credits Attempted and Earned 


Program Group 


Business-as- (Alternative 

Per-Student Outcomes Usual Group Placement) Difference 
Developmental education credits: 

Attempted 3.505 2.452 -1.053*** 

Earned 1.745 1.118 -0.627*** 
College-level credits in math/English: 

Attempted 6.095 6.350 02554" 

Earned 3.874 3.975 0.101 


NOTES: Estimation includes college fixed effects; testing cohort fixed effects; demographic indicators; income 
indicators; algorithm values. All groups, N = 12,971. One college set to zero. Credits over all semesters enrolled. 
**ED < 01, **p < .05, *p < .10. 
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The program effects used to measure effectiveness are identified as college-level 
credits earned in math and English. In fact, program group students had slightly lower pass 
rates in college-level math and English courses (62.6 percent vs. 63.6 percent, as calculated 
from Table 4.2); but because they attempted more courses, they accumulated more credits. 
As shown in Table 4.2, despite having attempted 0.255 more credits, program group students 
earned only 0.101 more credits (note that this earned credit result is not statistically 
significant). Overall, outcomes were higher for program group students than for business-as- 
usual group students. Although this gain in earned college-level credits is not statistically 
significant relative to the business-as-usual students, it may be economically significant as 
part of a cost-effectiveness analysis. 


Indirect Costs of Alternative Placement 


Program effects on credits attempted are used to calculate the indirect costs of the 
alternative placement system. When students take fewer developmental education courses, 
there are substantial savings for them. There is also much lower expenditure on 
developmental education by colleges (and society). However, there is an offsetting increase 
in spending on college-level courses. 


Indirect costs are the costs of all attempted credits in math and English. We calculate 
the cost per developmental credit and per college-level credit from Integrated Postsecondary 
Education Data System (IPEDS) data. For each of the sample colleges we calculate the cost 
per credit based on expenditures per FTE; this yields a cost per credit of $520 in 2016 dollars 
(results for each college are shown in Appendix Table A.33).* 


On average, the cost per developmental education credit is approximately equal to 
the cost per college-level credit. Per-credit costs are driven by two main factors: class size 
and faculty pay. For developmental and college-level classes, these factors are offsetting: 
Developmental classes are typically smaller than college-level classes, but faculty pay per 
class is lower. On net, the difference in cost between developmental and college-level classes 
is very modest. 


Funding per credit is divided between public support and student tuition/fees. For 
each of the sample colleges the division between these two groups is calculated from IPEDS 
data. Tuition/fees are calculated at 39 percent of total expenditure per credit (see Appendix 
Table A.33 for rates per college). Thus, students pay $200 per credit, and public funding 
covers the remaining $320. By regulation, developmental and college-level credits are both 


35 SUNY budget data yield similar estimates: tuition/fees for SUNY in-state residents at 
two-year colleges from SUNY (n.d.-a, n.d.-b ). 
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eligible for public funding.*° State funding regulations and federal loan aid eligibility are 
somewhat different with respect to developmental and college-level credits. Possibly, public 
funding of developmental credits may be incomplete, and/or students may face higher prices 
for developmental credits. For simplicity, we assume that developmental and college-level 
credits are funded equally from each source. 


As shown in Table 4.2, business-as-usual group students attempted 3.505 
developmental credits and 6.095 college-level credits on average; program group students 
attempted 2.452 developmental credits and 6.350 college-level credits. Accounting for the 
cost per respective credit, the indirect cost for business-as-usual placement was $4,990 and 
the indirect cost to the college for alternative placement was $4,580. These amounts should 
be added to the direct cost to derive the total college resources devoted to students under these 
placement systems. 


Cost-Effectiveness Analysis 


Given the results from the impact evaluation and cost analysis, the conclusion 
regarding alternative placement is straightforward. The alternative system is more cost- 
effective than the business-as-usual placement system: The alternative system was lower in 
cost and more effective from a social perspective. 


The per-student results from the social perspective are shown in Table 4.3. Critically, 
the total cost of alternative placement was $280 less than business-as-usual placement: 
Students took fewer developmental credits (saving $550) that more than offset the direct cost 
of alternative placement and the extra college-level credits (at $140 and $130 respectively). 
Alternative placement is more effective, with 0.101 more college-level credits earned after 
three terms. Thus, the cost per earned college-level credit is $1,300 for business-as-usual 
placement and $1,190 for alternative placement. 


36 In general, developmental courses are eligible for state aid (SUNY Chancellor’s Office, 2014). 
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Table 4.3 


Cost-Effectiveness Analysis: Social Perspective 


Business-as- Alternative 
Per-student Costs Usual Placement Placement Difference 
Direct cost: Placement $30 $170 $140 
Indirect cost: Attempted developmental $1,820 $1,280 $550 
credits 
Indirect cost: Attempted college-level credits $3,170 $3,300 $130 
in math/English 
Total Cost $5,020 $4,750 $280 
Earned college-level credits in math/English 3.874 3.975 0.101 
Cost per earned college-level credit $1,300 $1,190 -- 


SOURCES: Tables 4.1 and 4.2; authors’ calculations. Cost figures rounded to nearest 10. 


We undertake parameter-based sensitivity testing on the baseline results. We vary a 
series of key parameters: the program effects, the direct implementation cost, and the cost per 
credit (and hence the indirect cost). Each parameter is varied sequentially. The goal of the 
sensitivity analysis is to identify the conditions under which alternative placement converges 
to the cost-effectiveness of business-as-usual placement. The sensitivity tests show that the 
baseline conclusion is robust. Alternative placement is found to be more cost-effective than 
business-as-usual placement for all of the (plausible) alternative scenarios. The series of test 
results is shown in Appendix Table A.34. Even if alternative placement is less effective by 
one standard error, or as resource-intensive as the highest cost college, it is still more cost- 
effective than business-as-usual placement. 


This cost-effectiveness analysis adopts a societal perspective, looking at all costs and 
effects regardless of who is impacted. This perspective is appropriate for comparisons of cost- 
effectiveness in systems where funding formulae vary. However, two alternative perspectives 
— of students and of individual colleges — are important. 


For students, alternative placement is clearly cost-effective. Simply, students do not 
pay the additional costs of implementing the alternative placement system. From their 
perspective, alternative placement is more cost-effective if it leads to higher rates of credit 
accumulation in relation to the number of courses taken, that is, if alternative placement is 
more effective. From the student perspective, alternative placement is clearly more cost- 
effective than business-as-usual placement. For students the only cost was the tuition/fees 
they paid for credits attempted. As students took 0.798 fewer credits under the alternative 
system, they saved $160. However, as students generally do not want to take developmental 
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education courses, it may be more valid to focus on their developmental education savings 
from alternative placement. If students enrolled in 1.053 fewer developmental credits, they 
saved $210 in tuition/fees (4 percent of their total spending on college). 


For colleges, the determination of cost-effectiveness is less straightforward. Colleges 
must pay to implement the alternative placement system; this additional cost must then be 
recouped by increases in net revenues (revenues over costs) from additional coursework. 
However, the relationship between coursetaking and net revenues varies across colleges; it 
depends on college size, college capacity, and funding formulae. A numerical estimate of the 
cost-effectiveness ratio from the college perspective is therefore not possible. Nevertheless, 
given that alternative placement reduces total costs and increases credit accumulation, it is 
plausible to conclude that alternative placement is cost-effective from the college perspective. 


This economic analysis establishes a clear conclusion regarding the alternative 
placement system: It is more effective in terms of credits earned than business-as-usual 
placement, and it is lower in cost than the business-as-usual system (when direct and indirect 
costs are counted). Therefore, alternative placement is more cost-effective from both the 
social and student perspectives. 
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Chapter 5 
Conclusion 


This report is the second of three to emerge from a random assignment study of a multiple 
measures, data analytics placement system conducted at seven SUNY community colleges with 
students who entered in the fall 2016, spring 2017, and fall 2017 terms. In terms of impacts, the 
current report focuses primarily on outcomes of students in these cohorts in their first three terms 
of college enrollment; a future report planned for summer 2022 will examine outcomes over 
additional follow-up terms. The current report describes how the alternative placement system 
was developed and implemented, its costs, and its impacts, using a sample of 12,971 students 
who took a placement test at the seven colleges. Here we review some of the key findings from 
the study and discuss several implications and recommendations. 


Key Findings 


Implementation of the multiple measures, data analytics placement system was 
complex but successfully achieved. 


The design and implementation of the alternative placement system was considerably 
more complex than initially expected by both the research team and participating staff at the 
colleges. At the same time, every college was successful in overcoming barriers and 
developing the procedures needed to support the operation of the data analytics placement 
system. Five colleges achieved this benchmark in time for placement of students entering in 
the fall of 2016, while the other two colleges did so in time for new student intake in the fall 
of 2017. 


Both colleges and students experienced cost savings due to the use of the alternative 
placement system. 


From the college perspective, the total cost of using multiple measures, data 
analytics placement was $280 less per student than business-as-usual placement. 
Students attempted fewer developmental education credits (saving $550) that more than 
offset the direct cost of the alternative placement system and the additional college-level 
course enrollments that were provided (at $140 and $130 respectively). 


Students placed using the alternative system attempted an average of 0.798 fewer 
credits through three semesters, thus saving $160 in tuition/fees. However, as students 
generally do not want to take developmental education coursework, it may be more valid to 
focus on their developmental education course savings. Students attempted an average of 
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1.053 fewer developmental education credits, saving $210 in associated tuition/fees (4 
percent of their total spending on college). 


The multiple measures, data analytics system increased rates of placement into 
college-level courses, especially in English. 


Our main impact analyses consider the outcomes of three cohorts of students during 
three semesters following their initial placement testing. Many program group students were 
placed differently than they would have been under the status quo system. In math, 16 percent 
of program group students were “bumped up” to a college-level course (i.e., they were placed 
into college-level rather than developmental coursework); 10 percent were “bumped down” to 
a remedial course. In English, 44 percent were bumped up and 7 percent were bumped down. 


In math, alternative placement modestly increased the rate of placement into 
college-level math, but gains in college-level math enrollment and completion were small 
and short-lived. 


Alternative placement increased the rate of placement into a college-level math 
course by 6.5 percent. In terms of enrollment, program group students were 2.4 and 1.6 
percentage points more likely than their peers to take college-level math within one and two 
terms, respectively; however, the program group students were no more likely to enroll in a 
college-level math course than their business-as-usual peers within three terms. The initial 
small, positive impacts on the probability of completing (with grade C or higher) college- 
level math similarly faded over time. Although students placed by the algorithm were 2.0 
percentage points more likely to pass college-level math in the first term, through the second 
and third term there were no discernable differences between the two groups. 


Impacts in English were larger than those in math across all outcomes considered. 
Moreover, positive impacts in English were sustained through all three terms, although the 
magnitude of the gains declined over time. 


Program group students in English were 33.8 percentage points more likely than those 
in the business-as-usual group to be placed into a college-level English course. Program 
group students were also more likely to both enroll in and complete college-level English. 
Specifically, program group students were 12.7, 6.9, and 5.3 percentage points more likely to 
enroll in a college-level English course within one, two, and three terms of testing, 
respectively. And they were 6.3, 3.3, and 2.9 percentage points more likely to complete (with 
grade C or higher) a college-level English course over the same periods. 


Program group students earned slightly more college-level credits than business- 
as-usual group students. 
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While there is no evidence that the alternative placement system impacted student 
persistence or associate degree attainment, students in the program group earned, on average, 
slightly more college-level credits than students in the business-as-usual group. Program 
group students earned 0.35 and 0.31 more college-level credits within the first term and first 
two terms, respectively. Within three terms, program group students earned 0.24 more 
college-level credits, but this difference was no longer statistically significant. These gains 
were driven mostly by the algorithm’s impact on college-level coursetaking in English. 


With the exception of men in math, all demographic groups had better initial 
outcomes as a result of being placed using the alternative system. In English, these led to 
program group course completion rates through three terms that, compared to those of 
their same subgroup peers, were higher for women, Pell recipients, non-Pell recipients, 
and Black students. 


In math, all gender, Pell recipient status, and race/ethnicity subgroups considered 
except men had higher rates of placement into a college-level math course under the 
alternative placement system. Alternative placement reversed placement gaps between men 
and women but enlarged placement gaps between White students and Black and Hispanic 
students. Women, non-Pell recipients, and White students in the program group were 3.5, 3.8, 
and 3.2 percentage points more likely than their same subgroup peers in the business-as-usual 
group to complete (with grade C or higher) a college-level math course within one term of 
placement testing. But in terms of both enrollment and completion, by the third term there 
were no differences in rates for these outcomes between program and business-as-usual group 
students for any subgroup considered. 


In English, for all demographic subgroups considered, there were positive impacts on 
college-level English placement, enrollment, and completion outcomes under the alternative 
placement system. While these impacts declined over time, significant gains remained over 
three terms for some subgroups. Gaps between associated subgroups reversed in favor of 
women and narrowed in favor of Black students with regard to placement. Yet gaps between 
gender, Pell recipient status, and race/ethnicity subgroups were not reduced for the enrollment 
or completion outcomes. Nonetheless, by the end of the third term, women, Pell recipients, 
non-Pell recipients, and Black students in the program group had college-level English 
completion rates (with grade C or higher) that were 4.6, 4.5, 3.0, and 7.1 percentage points 
higher than their same subgroup peers in the business-as-usual group. 


Program group students who were bumped up through alternative placement 
from developmental to college-level courses had substantially better outcomes in both 
math and English. 
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Program group students whose placements changed to college-level from what their 
business-as-usual placements would have been experienced substantially greater benefits 
overall than did the full sample of program group students in the study, whose results were 
diluted by being grouped with students whose placements did not change (as well as being 
grouped with those whose placements changed from college-level to developmental, as will 
be discussed below). Bumped-up program group students were 8-10 percentage points more 
likely to complete (with grade C or higher) a college-level math or English course within 
three terms, compared to similar business-as-usual group students, those who were placed 
into developmental courses through business-as-usual placement but who would have placed 
into college-level courses under alternative placement (i.e., these business-as-usual students 
were also in the bump-up zone). 


Program group students who were bumped down through alternative placement 
had substantially worse outcomes in both math and English. 


Program group students whose placements changed to developmental from what their 
business-as-usual placements would have been experienced substantially worse outcomes 
compared to similar business-as-usual group students, those who were placed into college- 
level but who would have placed into developmental under alternative placement. This 
finding rebuts a common claim that students who are underprepared benefit from 
participating in developmental education courses. At least with regard to these students in the 
bump-down zone, who were near the cutoff for placement into college-level courses, 
placement into developmental education is found to be detrimental, on average. Bumped- 
down program group students were 8—10 percentage points less likely to complete a college- 
level math or English course than similar business-as-usual students, those who were placed 
into college-level courses through business-as-usual placement but who would have placed 
into developmental courses under alternative placement. 


Implications and Recommendations 


Colleges continue to seek ways to give students a good start in their higher education 
journey. The results of this study suggest that placement that uses multiple measures is one 
way to make sure that entering students are given a better opportunity to succeed in math and 
English, as well as in college generally. Some more specific lessons from this research are: 


As seen in other studies, single placement tests are not a good measure of student 
readiness to undertake college-level courses. High school GPAs, especially when combined 
with other measures, are a much better predictor of college-level course success. 
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In the process of analyzing historical data from seven colleges, we replicated findings 
of other studies (e.g. Belfield & Crosta, 2012; Scott-Clayton, 2012) that have demonstrated 
the limited value of single placement tests. While high school performance data can be 
difficult to obtain, they have significant value in predicting success in college-level courses. 
Colleges looking for ways to improve placement strategies should incorporate the high school 
GPA, in particular, as a key data point. 


Colleges would be wise to set up placement systems that allow more students into 
college-level courses. In this study, students who were close to college-ready were much 
better off if they were placed into college-level courses. This can be accomplished without 
negatively influencing course pass rates. 


Our analyses, especially those of bumped-up and bumped-down program group 
students (those whose placements changed due to use of the algorithm), suggest that 
placement into college-level math or English courses should be the default choice for students 
near the status quo placement cut scores. Students who were bumped up had considerably 
better outcomes than their business-as-usual group peers. Conversely, program group 
students who were close to the status quo placement cutoff score but who were placed into 
developmental courses (bumped down) were substantially more likely to experience poor 
outcomes than their business-as-usual group peers who placed into college-level courses. 
They were considerably less likely to succeed in gatekeeper college-level courses, even after 
three semesters. 


These findings have important implications for those setting cutoffs in any system 
of multiple measures. One concern often expressed by decision makers is that students 
may be harmed by being placed into college-level courses in which the material is too 
challenging for them. Our findings indicate that course pass rates of students bumped up 
into college-level courses were very similar to those of students placed using status quo 
methods. The status quo pass rate (with grade C or better) in math was 63 percent; the 
bumped-up pass rate was 60 percent. The status quo pass rate in English was 67 percent; 
the bumped-up pass rate was 65 percent. 


The use of a multiple measures placement system may contribute to better outcomes 
for underserved demographic populations, though it may do little to reduce equity gaps 
between related subgroups. 


In this study, all demographic groups considered — with the exception of men in 
math — benefited, at least initially, from placement using the multiple measures, data 
analytics system. And in English, women, Pell recipients, non-Pell recipients, and Black 
students in the program group had college-level English completion rates (with grade C or 
higher) that were substantially higher than their same subgroup peers over three terms. 
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However, enrollment and completion gaps between associated subgroups did not 
change very much. In some but not all cases, alternative placement reduced placement gaps 
in favor of women and traditionally underrepresented groups. But enrollment and completion 
gaps did not narrow (or widen) between associated subgroups over three terms. 


These findings differ from other studies in which students of diverse backgrounds 
have been permitted to begin their college careers in college-level math and English. For 
example, Hu, Park, Mokher, Spencer, Hu, & Bertrand Jones, (2019) studied Florida’s 2014 
policy that permitted most students to matriculate directly into college-level courses. They 
found that cohort-based passing rates in math and English courses increased, especially for 
Black and Hispanic students. Credits earned also improved, with Black and Hispanic students 
experiencing a greater increase than White students. 


The use of a better placement system is a positive step. However, more is needed to 
improve student outcomes. 


While students benefited from the use of a multiple measures, data analytics 
placement system, the effects diminished over time. These findings are consistent with 
studies of many other interventions which are found to generate positive outcomes that then 
fade with time. For example, a study of summer bridge programs in Texas found initial 
positive results, which were not sustained in students’ later semesters (Barnett et al., 2012). 


Many colleges are instituting reforms that are complementary to the use of alternative 
placement methods. In particular, corequisite courses are gaining in popularity. In these 
courses, students take college-level math and English courses along with a supplemental 
course in which they are provided with extra content and support needed to be successful in 
the college course. Studies show that participation in corequisite courses improves students’ 
outcomes (Cho, Kopko, Jenkins, & Jaggars, 2012; Logue, Watanabe-Rose, & Douglas, 
2016; Ran & Lin, 2019). 


There is also increasing interest among colleges in reorganizing the student 
experience in ways that (1) are comprehensive enough to change how students progress 
through their entire postsecondary education trajectory and (2) are broad enough that they 
affect most students in a college or at least most of those deemed in need of assistance to be 
successful. Two examples of broad-scale approaches are “guided pathways” reforms and the 
Accelerated Study in Associates Program (ASAP).*’ Both incorporate evidence-based 


37 For more information on guided pathways, see Jenkins, Lahr, Brown, & Mazzariello (2019). 
For more information on ASAP, see Miller, Headlam, Manno, & Cullinan (2020). Random assignment 
studies of ASAP have found that the program produces significant large effects on graduation in both New 
York City and Ohio. 
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practices that are organized to ensure that every student has a well-organized, supported 
educational experience (Barnett & Kopko, in press). 


The cost and effort to establish a data analytics placement system may not be worth it. 
Simpler and more affordable multiple measures assessment models have also proven effective. 


The multiple measures, data analytics placement system developed for this study 
required a relatively sophisticated analysis of each college’s historical student performance 
data to develop an algorithm that was then used to integrate information from multiple 
sources into each college’s human and technical infrastructure. In a study conducted in 
Minnesota and Wisconsin, similarly positive outcomes during the first semester of 
implementation were obtained using a simpler multiple measures placement system 
(Cullinan, Barnett, Kopko, Lopez, & Morton, 2019).** Colleges participating in that study 
developed a set of decision rules, informed by prior research and local knowledge, that were 
used to place students. Such an approach appears to require less effort than the approach used 
in the research described here. 


Final Thoughts 


This study sheds light on an important way to smooth the road for students entering 
college. Rather than using standardized placement tests alone, colleges can develop and 
deploy a multiple measures placement system that does a better job of assessing students’ 
readiness for math and English courses at a relatively low cost. The use of such a system, in 
tandem with other meaningful initiatives, can make a real contribution toward improving 
student success in college. 


38 More information on costs and impacts are expected in 2021. 
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Appendix Table A.1 


College Characteristics 


Institution 


Cayuga Jefferson Niagara Onondaga Rockland Schenectady Westchester 


Student population 7,001 5,513 7,712 23,984 10,098 8,458 22,093 
Full-time faculty 69 80 151 194 122 79 215 
Part-time faculty 170 177 0 480 409 0 2 
Student/faculty ratio 20 18 16 23 23 23 16 

% receiving financial aid 92 91 92 92 56 92 70 

Race/ethnicity (%) 

American Indian or Alaska Native 0 1 1 1 0 1 
Asian 1 2, 1 3 5 7 4 
Black or African American 5 7 11 12 18 14 21 
Hispanic/Latino 3 11 3 5 20 6 32 
Native Hawaiian or other 0 0 0 0 0 0 
White 85 73 80 49 39 67 33 
Multi-ethnic 2 3 2 3 2 2 2. 
Race/ethnicity unknown 3 3 1 27 15 2 5 
Non-resident alien 1 1 0 0 1 0 1 

Gender (%) 

Female 60 58 59 52 54 53 53 
Male 40 42 41 48 46 47 47 
Age (%) 
Under 18 30 17 19 24 10 37 1 
18-24 44 52 60 55 63 40 69 
25-65 26 31 21 21 26 23 30 
Age unknown 0 0 0 0 0 0 0 
Retention/graduation rates (%) 
Full-time students 56 55 63 37 68 56 64 
Part-time students 28 30 47 34 56 50 53 
Three-year graduation rate 24 27 28 20 29 20 15 
Transfer out rate 18 19 18 22 19 22 18 


NOTES: Based on fall 2015 Integrated Postsecondary Education Data System (IPEDS) data. 
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Appendix Table A.2 
Math Algorithm Components by College 


High Years Since HS Regents SAT ACCUPLACER ACCUPLACER 
School High School Diploma/ Math Math Arithmetic ACCUPLACER _ College-Level 
GPA Graduation — GED Status Score Score Score Algebra Score Math 
College 1 x x Xx x x x 
College 2 x x Xx x x x x x 
College 3 x Xx Xx x x 
College 4 
College 5 x x x x x 
College 6 
College 7 Xx x Xx x 
Appendix Table A.3 
English Algorithm Components by College 
High High Years Since HS ACCUPLACER _ WritePlacer or 
School School HighSchool Diploma/ ACCUPLACER _ Sentence Skills Other Writing 
GPA Rank — Graduation GED Status __ Reading Score Score Score* 
College 1 x Xx x xX x 
College 2 Xx x x x x x 
College 3 Xx x Xx x x 
College 4 x Xx x x x x x 
College 5 x Xx x x x 
College 6 x x x 
College 7 x x x x 
*To test writing skills, some colleges administered WritePlacer, an ACCUPLACER sub-test, while others administered a test created 
by the college. 
Appendix Table A.4 
Historical Severe Error Rates, by College 
College 1 College 2 College 3 College 4 College 5 College 6 College 7 
Overplaced 24.0% 5.7% 12.3% 11.2% 15.8% 
Math Underplaced 8.3% 44.7% 29.1% 36.0% 18.5% 
Severe error rate 32.3% 50.4% 41.3% 47.1% 34.3% 
Overplaced 12.0% 15.2% 13.7% 16.5% 8.4% 17.5% 10.7% 
English — Underplaced 30.7% 29.8% 33.7% 25.4% 43.7% 28.6% 40.4% 


Severe error rate 


42.7% 45.0% 47.5% 42.1% 52.1% 46.1% 51.1% 


NOTES: Calculations are based on a passing grade of B or better. 
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Appendix Table A.5 
Baseline Student Characteristics by College 


Business-as- 


Overall Program Usual College 1 College 2 College 3 College 4 College 5 College 6 College 7 
Std. Std. Std. Std. Std. Std. Std. Std. Std. Std. 
Characteristics Mean Dev. Mean Dev. Mean Dev. Mean Dev. Mean Dev. _Mean Dev. Mean Dev. _Mean Dev. _Mean__Dev. Mean ___Dev. 
Gender 
Female 48 50 48 50 47 50 58 49 54 50 53 50 48 50 52 50 55 50 40 49 
Gender missing (%) 5 22 5 22 5 22 0 4 0 5 0 0 0 6 0 2 0 0 13 34 
Race/ethnicity (%) 
American Indian/Native American 1 10 1 10 1 10 1 8 1 12 1 12 1 12 0 =) 1 8 1 10 
Asian 3 16 2 16 3 16 1 8 1 11 1 10 2 15 5 22 8 27 2 15 
Black 20 40 20 40 19 39 9 28 17 37 19 40 23 42 22 41 32 47 19 39 
Hispanic 20 40 20 40 19 39 5 21 3 17 4 20 11 31 28 45 13 34 33 47 
Pacific Islander 0 4 0 4+ 0 4 0 0 1 8 0 0 0 2 0 5 0 0 0 4 
White 43 49 42 49 44 50 81 39 69 46 55 50 53 50 36 48 41 49 24 43 
More than one race/ethnicity 3 18 3 18 4 19 1 11 3 17 4 20 6 23 3 17 3 18 3 17 
Non-resident alien 0 6 0 6 0 6 1 8 1 11 1 8 0 4 0 2 0 0 0 5 
Race/ethnicity unknown 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
Race/ethnicity missing 10 30 10 30 10 30 3 16 4 19 14 34 3 17 6 24 2 15 17 37 
Age 
Age at Test 21 6 21 6 21 7 21 6 23 8 22 7 20 6 22 7 25 9 20 5 
Age at Test Missing (%) 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
24 and Younger (%) 84 37 85 36 84 37 86 35 73 44 79 41 88 32 81 39 62 49 91 29 
25 and Older (%) 14 35 14 34 15 39 13 33 24 43 20 40 10 30 17 37 35 48 8 27 
Pell Grant 
Pell Grant recipient (%) 43 50 43 50 43 49 52 50 47 50 49 50 42 49 32 47 60 49 42 49 
Pell Grant recipient missing (%) 13 33 13 33 13 33 9 28 13 34 15 36 14 35 8 28 14 35 13 34 
Tested math (%) 75 43 76 43 74 44 98 13 87 33 90 30 0 0 97 16 0 0 94 24 
Tested English (%) 83 38 83 38 83 37 64 48 86 34 55 50 100 0 56 50 100 0 97 16 
Tested math and English (%) 58 49 59 49 57 49 62 48 74 44 45 50 0 0 53 50 0 0 91 28 
Enrolled any course (%) 86 35 85 35 86 39 89 31 85 35 82 38 85 36 91 28 81 39 85 36 
Total 12,971 6,589 6,382 695 1,247 1,884 2,013 1,875 505 4,752 
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Appendix Table A.6 


Post-Randomization Characteristics by Treatment Assignment 


Business-as- Program Treatment 

Characteristics Usual Mean Mean Difference P-value Observations 
Gender 

Female 49.85% 50.48% -0.6% 0.49 12,324 

Gender missing 4.95% 5.02% -0.1% 0.85 12,971 
Race/ethnicity 

American Indian/Native American 1.08% 0.96% 0.1% 0.48 12,971 

Asian 2.63% 2.47% 0.2% 0.57 12,971 

Black 19.34% 20.49% -1.2% 0.10 12,971 

Hispanic 19.10% 20.19% -1.1% 0.12 12,971 

Pacific Islander 0.19% 0.18% 0.0% 0.94 12,971 

White 43.64% 42.21% 1.4% 0.10 12,971 

More than one race/ethnicity 3.68% 3.25% 0.4% 0.18 12,971 

Non-resident alien 0.36% 0.38% 0.0% 0.86 12,971 

Race/ethnicity unknown 0.00% 0.00% 0.0% 12,971 

Race/ethnicity missing 9.98% 9.88% 0.1% 0.85 12,971 
Age 

Age at test 21.2 21.0 0.2 0.13 12,971 

Age at test missing (%) 0.00% 0.00% 0.0% 12,971 
Pell Grant 

Pell Grant recipient (%) 49.74% 50.45% -0.71% 0.46 11,101 

Pell Grant recipient missing (%) 12.75% 12.87% -0.1% 0.84 12,971 
TAP 

TAP Grant recipient 30.88% 30.76% 0.1% 0.88 12,971 

Missing TAP 12.75% 12.87% -0.1% 0.84 12,971 
GED 

GED recipient 8.19% 6.94% 1.3% 0.01 12,971 

GED recipient missing 0.00% 0.00% 0.0% 12,971 
GPA 

HS GPA (100 scale) 77.9 78.2 -0.21 0.22 8,021 

HS GPA missing 38.42% 37.91% 0.5% 0.55 12,971 
Accuplacer 

Algebra 51.0 50.9 0.16 0.74 9,643 

Arithmetic 46.3 46.6 -0.29 0.61 7,353 

Algebra 51.0 50.9 0.16 0.74 9,643 

College-level math 34.8 34.0 0.82 0.39 898 

Reading 72.2 71.7 0.52 0.22 10,359 
Total 

Sentence skill 75.9 75.6 0.30 0.58 5,114 

Writing 6.0 6.0 0.00 0.96 7,123 
Total 6,589 6,382 12,971 
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Appendix Table A.7 
Impact on Placement Into College-Level Math 
(1) 


College-Level Placement 


Program assignment 0.065*** 
(0.008) 

Business-as-usual mean 0.372 

Observations 9,693 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates 
control for full set of covariates as well as college and testing cohort fixed effects. 
***D < 01, **p < .05, *p < .10. 


Appendix Table A.8 


Impact on Enrollment in College-Level Math 


(1) (2) (3) 
1 Term 2 Terms 3 Terms 
Program assignment 0.024*** 0.016* 0.014 
(0.008) (0.009) (0.009) 
Business-as-usual mean 0.268 0.387 0.464 
Observations 9,693 9,693 9,693 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of 
covariates as well as college and testing cohort fixed effects. 
**ED < 01, **p < .05, *p < .10. 


Appendix Table A.9 
Impact on Completion of College-Level Math (With Grade C or Higher) 


(1) (2) (3) 
1 Term 2 Terms 3 Terms 
Program assignment 0.020*** 0.009 0.004 
(0.007) (0.008) (0.009) 
Business-as-usual mean 0.15 0.231 0.291 
Observations 9,693 9,693 9,693 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of 
covariates as well as college and testing cohort fixed effects. 
**ED < 01, **p < .05, *p < .10. 
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Appendix Table A.10 
Impact on Placement Into College-Level English 


(1) 


College-Level Placement 


Program assignment 0.338*** 
(0.008) 

Business-as-usual mean 0.463 

Observations 10,719 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates 
control for full set of covariates as well as college and testing cohort fixed effects. 
***D < 01, **p < .05, *p < .10. 


Appendix Table A.11 


Impact on Enrollment in College-Level English 


(1) (2) (3) 
1 Term 2 Terms 3 Terms 
Program assignment 0.127*** 0.069*** 0.053*** 
(0.009) (0.008) (0.008) 
Business-as-usual mean 0.444 0.615 0.661 
Observations 10,719 10,719 10,719 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of 
covariates as well as college and testing cohort fixed effects. 
**ED < 01, **p < .05, *p < .10. 


Appendix Table A.12 
Impact on Completion of College-Level English (With Grade C or Higher) 


(1) (2) (3) 
1 Term 2 Terms 3 Terms 
Program assignment 0.063*** 0.033*** 0.029% ** 
(0.008) (0.009) (0.009) 
Business-as-usual mean 0.28 0.396 0.442 
Observations 10,719 10,719 10,719 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of 
covariates as well as college and testing cohort fixed effects. 
**ED < 01, **p < .05, *p < .10. 
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Appendix Table A.13 
Impact on College-Level Credits Attempted 


(1) (2) (3) 
1 Term 2 Terms 3 Terms 
Program assignment 0.683*** 0.764*** 0.689*** 
(0.085) (0.161) (0.245) 
Business-as-usual mean 8.135 15.696 22.084 
Observations 12,971 12,971 12,971 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of 
covariates as well as college and testing cohort fixed effects. 
**ED < 01, **p < .05, *p < .10. 


Appendix Table A.14 
Impact on College-Level Credits Earned 
(1) (2) (3) 

1 Term 2 Terms 3 Terms 
Program assignment 0.34 7*** 0.311* 0.239 

(0.087) (0.165) (0.245) 
Business-as-usual mean 5.415 10.452 15.022 
Observations 12,971 12,971 12,971 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of 
covariates as well as college and testing cohort fixed effects. 
**ED < 01, **p < .05, *p < .10. 


Appendix Table A.15 
Impact on Continuous Persistence 
(1) (2) (3) 

1 Term 2 Terms 3 Terms 
Program assignment -0.007 -0.002 -0.006 

(0.006) (0.008) (0.008) 
Business-as-usual mean 0.811 0.614 0.44 
Observations 12,971 12,971 12,971 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of 
covariates as well as college and testing cohort fixed effects. 
**ED < 01, **p < .05, *p < .10. 
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Appendix Table A.16 


Impact on Associate Degree Attainment 


(1) (2) (3) 
1 Term 2 Terms 3 Terms 
Program assignment 0.000 0.000 -0.003 
(0.000) (0.001) (0.003) 
Business-as-usual mean 0.000 0.001 0.035 
Observations 12,971 12,971 12,971 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of 
covariates as well as college and testing cohort fixed effects. 
**ED < 01, **p < .05, *p < .10. 


Appendix Table A.17 
Impact on Placement Into College-Level Math (First Cohort Only) 


(1) 
CL Placement 


Program assignment 0.049% ** 
(0.012) 

Business-as-usual mean 0.408 

Observations 4,409 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates 
control for full set of covariates as well as college and testing cohort fixed effects. 
***D < 01, **p < .05, *p < .10. 


Appendix Table A.18 
Impact on Enrollment in College-Level Math (First Cohort Only) 


(1) (2) (3) (4) (5) 
1 Term 2 Terms 3 Terms 4 Terms 5 Terms 
Program assignment 0.024** 0.022* 0.006 0.010 0.013 
(0.012) (0.013) (0.013) (0.013) (0.013) 
Business-as-usual mean 0.299 0.406 0.499 0.531 0.547 
Observations 4,409 4,409 4,409 4,409 4,409 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of covariates as well as 
college and testing cohort fixed effects. 
***D < 01, **p < .05, *p < .10. 
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Appendix Table A.19 
Impact on Completion of College-Level Math With Grade C or Higher (First Cohort Only) 


(1) (2) (3) (4) (5) 
1 Term 2 Terms 3 Terms 4 Terms 5 Terms 
Program assignment 0.023** 0.012 0.002 0.013 0.018 
(0.011) (0.012) (0.013) (0.013) (0.013) 
Business-as-usual mean 0.166 0.243 0.314 0.341 0.36 
Observations 4,409 4,409 4,409 4,409 4,409 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of covariates as well as 
college and testing cohort fixed effects. 
***D < 01, **p < .05, *p < .10. 


Appendix Table A.20 
Impact on Placement Into College-Level English (First Cohort Only) 
(1) 


College-Level Placement 


Program assignment 0.342*** 
(0.014) 

Business-as-usual mean 0.466 

Observations 3,798 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates 
control for full set of covariates as well as college and testing cohort fixed effects. 
**ED < 01, **p < 05, *p < .10. 


Appendix Table A.21 
Impact on Enrollment in College-Level English (First Cohort Only) 


(1) (2) (3) (4) (5) 
1 Term 2 Terms 3 Terms 4 Terms 5 Terms 
Program assignment 0.178*** 0.098*** 0.080*** 0.072***  0.066*** 
(0.014) (0.013) (0.013) (0.012) (0.012) 
Business-as-usual mean 0.407 0.61 0.663 0.689 0.699 
Observations 3,798 3,798 3,798 3,798 3,798 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of covariates as well as 
college and testing cohort fixed effects. 
**ED < 01, **p < .05, *p < .10. 
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Appendix Table A.22 
Impact on Completion of College-Level English With Grade C or Higher (First Cohort Only) 


(1) (2) (3) (4) (5) 
1 Term 2 Terms 3 Terms 4 Terms 5 Terms 
Program assignment 0.114*** 0.07 1*** 0.068*** 0.063*** = 0.058*** 
(0.014) (0.015) (0.015) (0.015) (0.015) 
Business-as-usual mean 0.273 0.407 0.46 0.483 0.501 
Observations 3,798 3,798 3,798 3,798 3,798 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of covariates as well as 
college and testing cohort fixed effects. 
**ED < 01, **p < .05, *p < .10. 


Appendix Table A.23 


Impact on Continuous Persistence (First Cohort Only) 


(1) (2) (3) (4) (5) 
1 Term 2 Terms 3 Terms 4 Terms 5 Terms 
Program assignment -0.010 0.008 0.017 0.028** 0.017 
(0.009) (0.013) (0.014) (0.014) (0.012) 
Business-as-usual mean 0.831 0.648 0.489 0.397 0.25 
Observations 4,774 4,774 4,774 4,774 4,774 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of covariates as well as 
college and testing cohort fixed effects. 
**ED < 01, **p < .05, *p < .10. 


Appendix Table A.24 
Impact on Associate Degree Attainment (First Cohort Only) 


(1) (2) (3) (4) (5) 
1 Term 2 Terms 3 Terms 4 Terms 5 Terms 
Program assignment 0.000 0.001 0.002 0.002 -0.003 
(0.000) (0.001) (0.002) (0.006) (0.008) 
Business-as-usual mean 0.000 0.000 0.003 0.051 0.079 
Observations 4,774 4,774 4,774 4,774 4,774 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of covariates as well as 
college and testing cohort fixed effects. 
**ED < 01, **p < .05, *p < .10. 
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Appendix Table A.25 
Impacts on College-Level Math Outcomes by Student Demographic Subgroups 


Placement Enrollment Completion 
(1) (2) (3) (4) (5) (6) (7) 
1 Term 2 Terms 3 Terms 1 Term 2 Terms 3 Terms 
Gender 
Women only 
Program assignment 0.133*** 0.043 *** 0.023* 0.020 0.035*** 0.016 0.017 
(0.012) (0.012) (0.013) (0.014) (0.010) (0.012) (0.013) 
Business-as-usual mean 0.337 0.248 0.387 0.477 0.151 0.247 0.314 
Observations 4,600 4,600 4,600 4,600 4,600 4,600 4,600 
Men only 
Program assignment 0.001 0.005 0.009 0.007 0.006 0.002 -0.009 
(0.011) (0.011) (0.012) (0.012) (0.009) (0.011) (0.011) 
Business-as-usual mean 0.405 0.287 0.387 0.452 0.148 0.217 0.27 
Observations 5,093 5,093 5,093 5,093 5,093 5,093 5,093 
Pell Recipient Status 
Pell recipients only 
Program assignment 0.064*** 0.016 0.014 0.016 0.010 0.001 -0.001 
(0.012) (0.012) (0.014) (0.014) (0.010) (0.012) (0.013) 
Business-as-usual mean 0.322 0.262 0.403 0.494 0.140 0.228 0.293 
Observations 4,143 4,143 4,143 4,143 4,143 4,143 4,143 
Non-recipients only 
Program assignment 0.064*** 0.047 ** 0.03 1** 0.027* 0.038*** 0.023* 0.014 
(0.012) (0.013) (0.014) (0.014) (0.012) (0.013) (0.014) 
Business-as-usual mean 0.463 0.361 0.497 0.585 0.207 0.310 0.384 
Observations 4,154 4,154 4,154 4,154 4,154 4,154 4,154 
Race/ethnicity 
White students only 
Program assignment 0.1027*** 0.040*** 0.016 0.014 0.032*** 0.007 0.001 
(0.012) (0.013) (0.015) (0.015) (0.012) (0.013) (0.014) 
Business-as-usual mean 0.4 0.29 0.441 0.529 0.174 0.279 0.35 
Observations 3,879 3,879 3,879 3,879 3,879 3,879 3,879 
Black students only 
Program assignment 0.053*** 0.002 0.022 0.013 -0.006 -0.005 -0.014 
(0.019) (0.018) (0.020) (0.022) (0.014) (0.017) (0.019) 
Business-as-usual mean 0.287 0.231 0.33 0.414 0.118 0.186 0.246 
Observations 1,839 1,839 1,839 1,839 1,839 1,839 1,839 
Hispanic students only 
Program assignment 0.035** 0.014 0.007 0.002 0.004 -0.003 -0.008 
(0.015) (0.017) (0.018) (0.019) (0.016) (0.017) (0.018) 
Business-as-usual mean 0.419 0.354 0.473 0.557 0.195 0.275 0.334 
Observations 2,135 2,135 2,135 2,135 1,223 2,135 2,135 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of covariates as well as college and testing cohort 
fixed effects. 
**ED < O01, **p < 05, *p < .10. 
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Appendix Table A.26 
Impacts on College-Level English Outcomes by Student Demographic Subgroups 


Placement Enrollment Completion 
(1) (2) (3) (4) (5) (6) (7) 
1 Term 2 Terms 3 Terms 1 Term 2 Terms 3 Terms 
Gender 
Women only 
(0.012) (0.013) (0.013) (0.012) (0.013) (0.014) (0.014) 
Business-as-usual mean 0.461 0.451 0.645 0.698 0.302 0.432 0.478 
Observations 4,880 4,880 4,880 4,880 4,880 4,880 4,880 
Men only 
Program assignment 0.326% 0.120 0.064" 0.053% 0.055*** 0.023** 0.017 
(0.011) (0.011) (0.011) (0.010) (0.011) (0.012) (0.012) 
Business-as-usual mean 0.465 0.439 0.591 0.63 0.262 0.367 0.413 
Observations 5,839 5,839 5,839 5,839 5,839 5,839 5,839 
Pell recipient status 
Pell recipients only 
Program assignment 0.337% 0.168%** 0.095*** 0.073% 0.089% 0.045% 0.045%** 
(0.013) (0.013) (0.012) (0.011) (0.013) (0.014) (0.014) 
Business-as-usual mean 0.424 0.472 0.709 0.769 0.278 0.429 0.481 
Observations 4,617 4,617 4,617 4,617 4,617 4,617 4,617 
(0.012) (0.013) (0.012) (0.011) (0.014) (0.014) (0.014) 
Business-as-usual mean 0.536 0.580 0.746 0.795 0.387 0.509 0.565 
Observations 4,436 1,436 4,436 4,436 4,436 4,436 4,436 
Race/ethnicity 
White students only 
Program assignment 0.309% 0.098%** 0.052*** 0.044% 0.045% 0.011 0.005 
(0.012) (0.014) (0.013) (0.012) (0.014) (0.015) (0.015) 
Business-as-usual mean 0.512 0.517 0.691 0.728 0.369 0.503 0.546 
Observations 4,339 4,339 4,339 4,339 4,339 4,339 4,339 
Black students only 
Program assignment 0.397 0.201 *#* 0.130*** 0.096%" 0.092 0.071 0.071 
(0.018) (0.020) (0.019) (0.019) (0.018) (0.020) (0.020) 
Business-as-usual mean 0.345 0.385 0.59 0.654 0.19 0.302 0.355 
Observations 2,221 2,221 2,221 2,221 2,221 2,221 2,221 
Hispanic students only 
Program assignment 0.325% O.177#* 0.083*** 0.066% 0.082*** 0.036% 0.028 
(0.018) (0.020) (0.018) (0.017) (0.019) (0.021) (0.021) 
Business-as-usual mean 0.478 0.485 0.692 0.746 0.288 0.414 0.474 
Observations 2,195 2,195 2,195 2,195 2,195 2,195 2,195 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of covariates as well as college and testing 
cohort fixed effects. 
***D < 01, **p < .05, *p < .10. 
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Appendix Table A.27 


Impact on Placement Into College-Level Math (Students in Bump-Up and Bump-Down 
Zones Only) 


(1) 
College-Level Placement 
Bump-up zone 


Program assignment 0.921*** 
(0.010) 
Business-as-usual mean 0.007 
Observations 1,603 
Bump-down zone 
Program assignment -0.914*** 
(0.011) 
Business-as-usual mean 0.997 
Observations 950 
College FE YES 
Testing cohort FE YES 
Demographic indicators YES 
Income indicators YES 
Algorithm values YES 
Group All 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full 
set of covariates as well as college and testing cohort fixed effects. 
**ED < 01, **p < .05, *p < .10. 


Appendix Table A.28 
Impact on Enrollment in College-Level Math (Students in Bump-Up and Bump-Down 
Zones Only) 
(1) (2) (3) 
1 Term 2 Terms 3 Terms 
Bump-up zone 0.389% 0.270*** 0.219%*** 
Program assignment (0.018) (0.022) (0.023) 
Business-as-usual mean 0.052 0.267 0.373 
Observations 1,603 1,603 1,603 
Bump-down zone -0.420*** -0.258*** -0.210*** 
Program assignment (0.027) (0.028) (0.027) 
Business-as-usual mean 0.603 0.684 0.721 
Observations 950 950 950 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of 
covariates as well as college and testing cohort fixed effects. 
**ED < 01, **p < .05, *p < .10. 
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Appendix Table A.29 


Impact on Completion of College-Level Math With Grade C or Higher (Students in 
Bump-Up and Bump-Down Zones Only) 


(1) (2) (3) 
1 Term 2 Terms 3 Terms 

Bump-up zone 0.206*** 0.133*** 0.096*** 

Program assignment (0.016) (0.020) (0.022) 

Business-as-usual mean 0.031 0.171 0.257 

Observations 1,603 1,603 1,603 
Bump-down zone -0.150*** -0.111*** -0.087*** 

Program assignment (0.023) (0.027) (0.029) 

Business-as-usual mean 0.234 0.308 0.356 

Observations 950 950 950 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of 
covariates as well as college and testing cohort fixed effects. 
**ED < 01, **p < .05, *p < .10. 


Appendix Table A.30 
Impact on Placement Into College-Level English (Students in Bump-Up and Bump-Down 
Zones Only) 
(1) 
College-Level Placement 
Bump-up zone 
Program assignment 0.928% ** 
(0.005) 
Business-as-usual mean 0.051 
Observations 4,664 
Bump-down zone 
Program assignment -0.964*** 
(0.010) 
Business-as-usual mean 1 
Observations 746 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates 
control for full set of covariates as well as college and testing cohort fixed effects. 
**ED < 01, **p < .05, *p < .10. 
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Appendix Table A.31 
Impact on Enrollment in College-Level English (Students in Bump-Up and Bump-Down 


Zones Only) 
(1) (2) (3) 
1 Term 2 Terms 3 Terms 
Bump-up zone 0.339% 0.187*** 0.154*** 
Program assignment (0.012) (0.013) (0.012) 
Business-as-usual mean 0.278 0.516 0.573 
Observations 4,664 4,664 4,664 
Bump-down zone -0.395*** -0.171*** -0.163*** 
Program assignment (0.034) (0.032) (0.031) 
Business-as-usual mean 0.688 0.763 0.795 
Observations 746 746 746 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set of 
covariates as well as college and testing cohort fixed effects. 
**ED < 01, **p < .05, *p < .10. 


Appendix Table A.32 


Impact on Completion of College-Level English With Grade C or Higher (Students in 
Bump-Up and Bump-Down Zones Only) 


(1) (2) (3) 
1 Term 2 Terms 3 Terms 
Bump-up zone 0.186*** 0.104*** 0.087*** 
Program assignment (0.012) (0.013) (0.014) 
Business-as-usual mean 0.173 0.334 0.388 
Observations 4,664 4,664 4,664 
Bump-down zone -0.176*** -0.109*** -0.075** 
Program assignment (0.030) (0.034) (0.035) 
Business-as-usual mean 0.313 0.382 0.405 
Observations 746 746 746 


NOTES: Heteroskedastic robust standard errors in parentheses. Impact estimates control for full set 
of covariates as well as college and testing cohort fixed effects. 
**ED < 01, **p < .05, *p < .10. 
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Appendix Table A.33 


Financial Information for Colleges in Cost Analysis 


Cost per Credit Tuition/Fees 
College (Business-as-Usual) (%) 
A $510 41% 
B $710 32% 
C $460 42% 
D $470 41% 
E $510 38% 
F $570 38% 
Cohort-weighted average $520 39% 


SOURCE: IPEDS, 2017 academic year. 
NOTES: Cohort-weighted average based on total revenues and enrollments per college. 2016 dollars. 


Appendix Table A.34 


Parameter-Based Sensitivity Testing: Cost-Effectiveness Ratios 


Business-as- Alternative 

Per-student Parameters Usual Placement Placement 
Baseline cost-effectiveness ratio $1,300 $1,190 
Program effects on college-level credits attempted 

(+1 standard error for APS) $1,300 $1,200 
Program effects on college-level credits earned 

(—1 standard error for APS) $1,300 $1,210 
Direct cost (highest cost college) $1,300 $1,230 
Indirect cost per credit (highest cost college) $1,150 $1,050 


SOURCES: Baseline results from Tables 4.2 and 4.3. Program effect standard errors from analysis in this 
report. Direct and indirect cost from Table A.33. 
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Appendix B 
Technical Notes on Analytic Method 
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To test the hypothesis that a multiple measures, data analytics placement system 
differs from a single test placement system, we conducted an intention-to-treat analysis by 
comparing the average outcomes for students assigned to the program and business-as-usual 
groups. Specifically, we estimated the following ordinary least squares (OLS) regression: 


[B.1] Y¥ij = a + PRY + AQ; + NiXi + 542) + Ej 


where Yj are academic outcomes for student 7 within j terms of taking the placement 
test; Rj indicates whether the individual was randomly assigned to be placed using the 
predictive algorithm; qj, is an indicator for the institution at which the student took their 
placement test; X; is a vector of baseline covariates including gender, race/ethnicity, age, 
and financial aid status; Zi includes both math and English algorithm calculations for each 
student (which are essentially two indices for academic preparedness); and ¢j is a random 
error term. The coefficient of interest is B, the effect of assignment to the new placement 
strategy on each outcome of interest. 


Importantly, the random assignment procedure should, in expectation, ensure that 
students assigned to the program group are similar in all ways to those assigned into courses 
under status quo placement rules. Table A.6 in Appendix A provides evidence that 
participants’ demographic and academic characteristics, including indicators for missing 
characteristics, are well balanced across program and business-as-usual groups for the final 
analytic sample. Students’ individual ACCUPLACER test scores also are similar across 
both groups. In order to avoid post-treatment bias, we report on and use only the student’s 
first ACCUPLACER score received on each subject test. With few exceptions, the 
differences between program and business-as-usual groups are small and statistically 
insignificant, providing reassurance that randomized intervention assignment was 
implemented as intended. Although students in the program group are slightly less likely to 
be White and more likely to be Black (p < .1), impact regressions include race/ethnicity 
covariates to control for unbalance between the program and business-as-usual groups. 
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Appendix C 
College-Level Math and English Credit Outcomes 
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Figure C.1 reveals limited impacts on the number of college-level math credits 
attempted and earned. Program group students attempted only 0.05 additional college-level 
math credits within one term of testing (p < .1), and by the second term there were no 
observable differences between students according to how they were placed into math 
courses. Small positive impacts on college-level math credits earned similarly faded over 
time; although program group students earned 0.07 more college-level math credits than the 
business-as-usual group students within one (p < .01) and two terms (p < .1) of testing, no 
discernable differences remained through the third term. 


Appendix Figure C.1 
College-Level Math Credit Outcomes (Among Students in the Math Subsample) 


2.55 2.58 


Term 2 Term 2 
Attempted Earned 
= Business-as-usual group = Program group 


#8 < OL, **p < 05, *p < .10. 


Figure C.2 shows that students who were placed by the algorithm attempted and 
earned more college-level English credits than their business-as-usual peers. The positive 
impacts were sustained across all three terms considered, and interestingly, the gains remained 
relatively stable. Specifically, in the first term following testing, program students attempted 
0.38 credits and earned 0.22 credits more than business-as-usual students (p < .01). Through 
the third term, the difference in credits attempted and earned between both groups was 0.34 
and 0.17 credits, respectively (p < .01). This may suggest that the multiple measure placement 
system’s impact on college-level English credits can be explained by program group students’ 
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earlier enrollment and completion of the initial required college-level courses and that these 
gains do not lead program group students to pursue additional, higher-level English courses 
at a different rate, at least within the first three terms following testing. 


Appendix Figure C.2 


College-Level English Credit Outcomes (Among Students in the English Subsample) 


Term 2 Term 2 
Attempted Earned 
= Business-as-usual group = Program group 


p< 01, **p < .05, *p < 10. 
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